i checkouted some libraries which did not log, what if i need some log info to debug those libraries?
i am using nimble to manage dependency at now, if i need manually add logs in specific line of a library then i think i'd better to use git submodule/subtree instead.
what i currently need are:
the std/logging says "If this library does not fulfill your needs, write your own.", but if i made one, how can i convince the community use it? not to mention i am new comer. thus i think that's the std/logging's job
with regards to “convince”, you make a good library and advertise it then people will use it.
what’s wrong with manually editing the libraries you need info from? You don’t really have the choice to otherwise I believe
Generally libraries don't log because logging is expensive. This might matter less in Python, which is a lot slower than Nim, but since people tend to use Nim at least partially for its high performance you generally don't want to log too much. Some libraries (like Jester) will log in a debug build, but not in a release build. This is done with compile-time switches so the code for logging doesn't even exist in the release build. And of course in this specific library the logging is very rudimentary.
In general though it seems like not a lot of people using Nim tend to use logs. Why is anyone's guess, but between the strong type system catching compile-time bugs and the fairly "obvious" nature of your typical Nim program I'd wager they're simply not needed. More complex libraries like Polymorph, which alters the control flow of your program, tend to provide more logging though.
When it comes to adding logging to existing libraries I'd probably use `patchFile` in your Nimble file (which is NimScript). What that will do is override files in imported modules with a file you supply. This way you can copy a file from a library you're using, modify it in your repository, and then patch that back into the actual dependency when you're compiling. This of course requires that you define your dependencies very strictly so you don't accidentally break the library your using.
Tha "If this library does not fulfill your needs, write your own" is mostly to deter people from trying to add too much stuff into the standard library logger. Of course you don't actually have to write your own library, I for one would recommend the chronicles library from Status. If you need more than that I'm not ever sure what kind of crazy stuff you would need.
IMHO libraries simply shouldn't log. If the library needs to "log", it should instead take a callback that clients can setup. Extra points if the callback is typed.
Not like this:
type InfoCallback* = proc (message: string) {.closure.}
proc someApi*(args; cb: InfoCallback)
But like this:
type
InfoKind = enum
crawledUrl,
httpError,
outOfBusiness
Info = object
case kind: InfoKind
of crawledUrl: url: string
of httpError: errorCode: byte
of outOfBusiness: reason: string
InfoCallback = proc (info: Info) {.closure.}
proc someApi*(args; cb: InfoCallback)
Benefits:
thanks for the operatable reply!
it takes me some time to understand. i like this "library-specific logger/callback" approach and i can utilize std/logging meanwhile, since the callback is library specific, i can put some context in it. structured info is also a good point.
Hmm, I like the idea of configurable "logging", but I'd much rather be dealt with without having to pass closures all over the place (annoying to pass a closure to every procedure from a library). What if we built a compile-time type based system. Something like:
superlog.nim:
import tables, macros
var activeLoggers {.compileTime.}: Table[string, NimNode]
macro log*(x: typed): untyped =
if activeLoggers.hasKey(x.getTypeInst.repr):
var
log = activeLoggers[x.getTypeInst.repr]
infoIdent = newIdentNode("info")
result = quote do:
block:
let `infoIdent` = `x`
`log`
else:
result = newStmtList()
macro registerLogger*(x: static[string], y: untyped): untyped =
activeLoggers[x] = y
module1.nim:
import superlog
type
Info* = object
msg*: string
AnotherInfo* = object
msg*: string
proc someProc*(task: string) =
echo task
log(Info(msg: "Task \"" & task & "\" failed succesfully"))
log(AnotherInfo(msg: "This message won't ever be added to the code"))
echo "Done"
module2.nim:
import superlog
registerLogger("Info"):
echo "Message: ", info.msg
import module1
someProc("greet")
Which would do this:
$ nim c -r module2
Hint: used config file '/home/peter/.choosenim/toolchains/nim-1.6.2/config/nim.cfg' [Conf]
Hint: used config file '/home/peter/.choosenim/toolchains/nim-1.6.2/config/config.nims' [Conf]
Hint: gc: refc; opt: none (DEBUG BUILD, `-d:release` generates faster code)
9878 lines; 0.014s; 8.641MiB peakmem; proj: /tmp/loggingtest/module2; out: /tmp/loggingtest/module2 [SuccessX]
Hint: /tmp/loggingtest/module2 [Exec]
greet
Message: Task "greet" failed succesfully
Done
Of course a little more polished, preferably without passing a string (maybe something like registerLogger(module.Info) and it would read out the identifiers. Unfortunately it can't import the module first and just pass a reference to the type, because that would expand the log macro before the registration happens.
Not having logging and application metrics is a frequent reason for large companies / FAANGs to fork or rewrite popular libraries, in my experience. Good metrics and logs are crucial in many environments. And application metrics are becoming more popular.
Of course it's always up to the main module to set up handlers and configure how logs/metrics from modules are treated - but this requires a standard interface in stdlib.
https://github.com/status-im/nim-chronicles supports similar ideas around compile-time enabled logging and loggers, removing the logging code when not used (with lots and lots of fancy options and ways to create compile-time configurations).
In practice however, compile-time based logging doesn't work very well, for several reasons:
what is really useful about chronicles is its support for structured logging, and the smooth syntax it has for achieving it - when outputting JSON to an ELK stack, you can run analysis on logs over time etc which is a great debugging help.
Regarding logging vs API richness, it really depends on the type of library - API richness is good for a number of reasons but often onerous to get right - logging provides a shortcut and is excellent for "business logic" (as opposed to abstract data types, algorithms etc).
Often, you might start out with logging then slowly push the logging out of the core logic and into the UX/busniess logic part of your application.
I agree with all of this, which is why I've used Chronicles in the past, but seen myself not including it in newer projects. However Chronicles and this Superlog concept has a few key differences in which Superlog seems to address most of your ire with compile-time logging. Let's go through the points one by one:
when you introduce logging, the public API tends to suffer - logging creates an "out-of-band" channel through which information is returned and if you also return it in the API, you've created API duplication, which rarely ends well (two ways to achieve the same result)
This would be the same in Chronicles and Superlog, but for that matter any structured logger. The concept of a flexible structured logger is based around the idea of passing data into the logger so it can be structured, instead of composing strings and passing those in. This is inherently creating an out-of-band communications channel, but if you want structured logging this is the time you have to pay. Of course you could force the loggers to be pure functions, at least this will force them to not be able to modify the global state of your program (apart from some kind of state for the logger which must be passed along to the logger).
logging is a side effect: if you compile-time-disable logging, you're creating a different execution flow for the non-logging version of the library - at first, this seems like a great idea, but soon these side effects become significant to the correct function of the application / library, and you end up not using the fancy logging magic you spent so much time on perfecting - it instead becomes a source of bugs and frustration
This is the first part where Chronicles and Superlog differs. Chronicles uses configuration to turn on and off logging, and to define how to log. Superlog on the other hand uses code to control the logging flow. This means that you really only have one version of your program, as long as you don't add any when defined switches yourself. And even if you add your own switches you still only have your own specifically defined scenarios to handle, not the explosion of potential versions of the code the mass of configuration options Chronicles offers.
you create multiple versions of your codebase - testing becomes difficult because now you have multiple versions to test (does the code still compile with "trace" logging? does it work with json? etc)
See the above, essentially Superlog doesn't create more versions of your code than you specify yourself with your own compile-time switches. The idea being that logging is actually part of the application and codebase, and not something that's switched on and off by compile-time flags. As you say testing all the constellations of possible options quickly become unwieldy, but testing with one or two custom defined switches is much easier. The only downside is that if you want to support JSON logging/database logging/carrier pigeon logging you have to add a line or two of code to your project which actually writes out JSON/writes to a database/feeds the pigeons.
your users come to expect logging to work in a particular way - however, relying on imports, or even worse, order of imports, for log formatting in general is a really, really bad idea because the formatting will be different depending on which imports you remembered to make
This isn't an issue with Superlog, users won't expect anything from logging, because there is no default behaviour (apart from removing all log statements). The only logging that is done is that which is done by the user themselves, and that is only based on the types the library exposes. Formatting won't be different if you don't change your code, no matter the order of imports. The only thing you need to keep in mind is importing Superlog and register your loggers before importing any modules you want to log from. But if you don't it should be immediately obvious as you won't get any logs at all.
your users typically really want to configure logging for themselves without recompiling the application - this requires a certain amount of "runtime" behavior, and again, all that fancy compile-time magic you invented is .. lost
I start to sound a bit like a broken record at this point, but since all the actual logging effort is done by the user it's completely up to the user to define which runtime behaviour they want. Run in a constricted environment and don't want to log anything? Remove it at compile-time. Building a GUI application? Maybe show some information as pop-ups, and write some other stuff to a text-log. Building a cool web-server? Sure, throw your logs into a database and build an analytics tool with them. Or maybe a neat terminal application? Add in your own verbosity switch and use that to select what to log.
In my current PoC I added a severity flag to log-lines, however in my new testing version I'm going to remove this and rather encode it in the default generic types. The idea of Superlog is that the libraries used shouldn't really have any say or idea about what is being logged (I have added a template which returns true/false for whether or not a type is logged, but that is only intended to remove expensive operations required to produce log output). Each library simply offers up types for what is loggable (with compile-time versions for each module, so you can say you want to log all Exceptions from the tables module, but not from the strutils module). The user application then starts off by specifying logger procedures before importing all the libraries and running the main code. This means the user is always 100% in control of what is being logged and how. And libraries which only lives in a single domain (say for example microcontroller programming) can define a set of lighter or heavier types for logging. When the severity flag is removed, there is nothing stopping you from creating an empty type and "log" that to simply notify the parent application that some part of the code was reached and it can "log" this however it wants (blink an LED for example). Since everything is type based you can even create hierarchies of types and log at any specificity you want (similar to how you can catch an exception or a specific version of an exception).
logging creates an "out-of-band" channel through which information is returned and if you also return it in the API, you've created API duplication
The out-of-band aspect of metrics and logging is its main feature: the information is consumed by somebody who is not the API caller.
This is why metric/logging libraries should never block or raise exceptions and be invisible to the effect system. You should be able to log and generate metrics from purely functional code.
I'm not a purist on this. If I'm using a private library, even if shared between projects, I'll often put in logging.
But for public libraries I never do logs. (Or the logs are fully behind some hidden development when clauses.)
Nim, as a language, is one that uses the often-artificial concept of thrown errors. (As opposed to in-line error handling used by some other languages.) This means that, philosophically, one should avoid secondary I/O effects, such as logging, at all costs. IMO, the user is assuming that a thrown exception when in a library is the result of not using the library correctly; perhaps bad input. They want to capture and control the logging from outside the library in their own code. While loggers do attempt to avoid raising exceptions, you can't ever really be sure of it.
(BTW, there is nothing wrong with Nim using thrown exceptions for error-handling. Since it goes to C, maintaining that methodology totally makes sense.)
That's one of the reasons I like the idea behind superlog. Essentially it allows you to unify all the when defined(myLibraryLog) switches into one common system, and allows all the logged information to follow a coherent pattern or target the same output.
What you mention with exceptions is a good point, and fortunately we have a system which tracks which exceptions can be raised by a procedure, and a way to force procedures to not raise any exceptions. It would be trivial to specify that all the logger callbacks should simply not be able to raise anything. If the system is also built around a state type that would be automatically passed in to each log call it would even allow the callbacks to be pure functions (since they can change arguments passed in as var). The only standing issue is the one of blocking. This could be solved by adding some utility procedures to make it easier to do it right, but I still feel like the core concept should always be able to give you zero-cost logging if you so desire.