My first serious project was Chrono a Timestamps, Calendars, and Timezones library for Nim.
https://github.com/treeform/chrono
I have become much better at Nim than when I first wrote it. I recently went over the project and wanted to share things that I have improved.
1. Github actions. Github actions are easy to set up and make sure your project/library stays tested. I now use this as a standard for all my Nim projects: https://github.com/treeform/chrono/blob/master/.github/workflows/build.yml It just runs nimbe test which means it will probably work for your project too without changes.
nim c -r tools/generate.nim json --startYear:2010 --endYear:2030 --includeOnly:"utc,America/Los_Angeles,America/New_York,America/Chicago,Europe/Dublin"
Some things have not changed and I still firmly stand by on:
D. Adding a timezone to a calendar is complex. There needs to be two functions … I call apply and shift. Both functions will make your calendar have the new timezone offset.
applyTimezone: "1970-05-23T21:21:18Z" -> "1970-05-23T14:21:18-07:00"
shiftTimezone: "1970-05-23T21:21:18Z" -> "1970-05-23T21:21:18-07:00"
I did not want to use json. I wanted to use binary formats .. compressed binary formats.
Bummer, I still like them better than json. :-)
I wrote one, called Fleece <https://github.com/couchbaselabs/fleece>. It's used in the Couchbase Lite database, because it's faster than JSON. There's now a Nim binding of it ... it's part of the experimental Couchbase Lite binding I did over the past week. It's not easily separable yet, and it requires the C++ library, but that can be cleaned up.
For a time library, the lowest building block should be the timestamp of a single float64, not a complex calendar object. You should store timestamps and transfer timestamps.
Yup. In units of seconds since 1/1/1970, right? Although I've sometimes worried how soon the precision will degrade, since there are 'only' 52 bits of mantissa.
There are cases where you want to store a (timestamp, timezone) tuple, like for blog posts, when it's useful to know what the local time was.
5. Always put your code into a src dir. Even though nimble supports not having src dir, it’s just better in every way.
What's the difference between putting my source code files into a src folder instead of putting them into my root folder? I'm not so familiar with nimble, but isn't nimble forcing (at default) to put all my files into src folder. Also if I have other dependencies like assets. I don't want to putt a assets folder or something similar into a src folder. And i do not see any benefits except if the project becomes huge. And if the project becomes huge, then I setup a project structure by myself I would like.
5. Always put your code into a src dir. Even though nimble supports not having src dir, it’s just better in every way.
In my experience that's the best way to break your project because on install nimble removes the src directory and then all your paths are wrong.
This is so broken that even in nimble you had workaround like this: https://github.com/nim-lang/nimble/blob/3ba8bd94/nimble.nimble#L4-L9
when fileExists(thisModuleFile.parentDir / "src/nimblepkg/common.nim"):
# In the git repository the Nimble sources are in a ``src`` directory.
import src/nimblepkg/common
else:
# When the package is installed, the ``src`` directory disappears.
import nimblepkg/common
for the same reason as ppl don't use floating points for banking applications, but instead use either integer in some units eg cents or fraction of cents, or other fixed point arithmetics. floats will cause issues with rounding or difference wrt roundtrips, not to mention stringification
I'd instead use: distinct int64 representing unix timestamp in either milliseconds (range of +/- 300M years) or microseconds (range of +/- 300K years); this will need BigInt for js platform. Caveat: leap seconds.
note: if your application requires large range/precision, then use: distinct int128 (doesn't exist yet apart from compiler/int128.nim) or type Timestamp = array[2, int64]; this could be exposed in nim and work with both js, c, vm backends (but raises some concern regarding whether we'll then need compiler/int256.nim; hopefully something smarter can be used as that'd be wasteful)
I did not want to use json. I wanted to use binary formats .. compressed binary formats.
that's very application dependent; if you need fwd/backward compatibility, or a schema, or performance, json is bad, protocol buffer or capnproto would be better
Write good tools with command line parameters
ideally, "everything is a library", but yes, sometimes a binary that wraps it is good
Always put your code into a src dir.
I agree.
src is just good hygiene that prevents bugs (eg prevents import pkg/cligen/oops_not_intended_as_source) and should be the default, if not enforced.
In my experience that's the best way to break your project because on install nimble removes the src directory and then all your paths are wrong. [..]This is so broken that even in nimble you had workaround like this
The issue you mention is real, and the workaround is bad; IMO the fix is to improve nimble, or, if it can't be fixed, add an import syntax import this/nimblepkg/common
Bummer, I still like them better than json. :-)
String stream only started to work with JS recently and still can't do pointer stuff. I wish binary formats was better supported in JS.
There are cases where you want to store a (timestamp, timezone) tuple
Exactly, I view timestamp as kind of a string and timezone kind of language code. If you need to store what language a string is in, yes store it. But most of the time that is not needed, and you just need to display time in the user's "language."
nimble removes the src directory
I never had an issue with that. Maybe you need to use staticRead instead of readFile?
float64 for timestamp
I still will stand for float64 timestamp for my application. Yes there are special places like banking that don't use float points. They are special. You would not use float points for money too. For almost every one else float64 since 1970 fits the bill.
Here is what I deal with on daily bases
- python 1.0
- java 1000
- js 1000
- bigquery 1000_000
- go 1000_000_000
That is why I use float64 seconds from 1970 utc.
leap seconds
I should add a j2000 mode as that will enhance my https://github.com/treeform/orbits library.
floats will cause issues with rounding or difference wrt roundtrips, not to mention stringification
I don't think the same issues apply to timestamps. They're not user-visible in raw form, and you're not usually working with adjacent timestamps that are nanoseconds apart, so it doesn't matter if they're off by some tiny epsilon. I like the fact that you never have to worry about overflows. And treeform's point about lack of int64 in JS is important — a lot of JS parsers, in various languages, just parse any number to float64.
C++'s std::chrono module does support arbitrary backing types for times (it's a template parameter) but IMHO that contributes to making the library super awkward to use; you have to invoke a manual conversion function to cast times to different units.
I don't think the same issues apply to timestamps
the moment you're using FP as your internal internal representation you're affected by FP semantics and the machine epsilon, eg:
var a = 1.2
var b = a + 1e14
var c = b - 1e14
echo (a, c, a == c)
even with a 10% range of variation you still can't rely on timestamp equality:
var a = 0.1
var b = a + 1
var c = b - 1
echo (a, c, a == c)
There's a good reason almost all datetime libraries use fixed point/integral arithmetics internally to represent time instead of FP: C (ctime), C++ (std::chrono, boost date_time), mongodb (int64 milliseconds since epoch), D (https://dlang.org/phobos/std_datetime_systime.html), and in particular nim: Time = (secs: int64,nsecs: int)
IMHO that contributes to making the library super awkward to use; you have to invoke a manual conversion function to cast times to different units.
that's a C++ problem; nim doesn't have this problem. Look at the std/times module which abstracts the internal representation as (secs,nsecs)
float may be easier on first sight from implementer point of view (not for user point of view), but it just causes more problem.
I even added fromUnixFloat+ toUnixFloat in https://github.com/nim-lang/Nim/pull/13044 so you can convert your user FP timestamps into internal std/Time (int64,int) representation and then just deal with internal representation for all operations.
It gives you nanosecond precision within +/- 300 billion years, so you can represent any nanosecond in the universe's timespan and well beyond, without any loss or catastrophic cancellation issues, and reliable timestamp equality.
for js, BigInt is IMO the right approach; i'ts supported in almost all browsers except for IE (which is EOL'd anyway) and safari (which is "hopefully almost there not giving up hope", see https://bugs.webkit.org/show_bug.cgi?id=179001 which has almost all dependent issues fixed); and there are polyfills eg https://github.com/peterolson/BigInteger.js
nimble removes the src directory
Wow, wasn't even aware of that. Why? The .nimble file with the srcDir directive is right there.
that's a C++ problem; nim doesn't have this problem. Look at the std/times module which abstracts the internal representation as (secs,nsecs)
For me Nim does have this problem. That is why I don't use std/times module. It just works badly in JS mode.
Yes in theory a more accurate time representation is best. But in practice the float64 wins out because it's easy to use and an interoperate between many systems.
good reason almost all datetime libraries use fixed point/integral arithmetics internally to represent time instead of FP
I don't think this is true. As I deal with float64 timestamp system all the time, its just it's usually in milliseconds because of javascript/java or seconds if I am dealing with python.
I don't think this is true
it is true across the board; maybe there are some FP time libraries out there, but the vast majority uses integral arithmetics over some unit (s, ms, us, ns are typical)
a = new Date(1e17)
Invalid Date
new Date(1.2).valueOf() 1 # => not 1.2! gets rounded to millisecond resolution
see https://stackoverflow.com/questions/51691164/how-does-javascript-date-gettime-milliseconds-map-to-a-64-bit-float and https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Numbers_and_dates