Hi all, I got a basic kernel for jupyter notebook working, you can find it on github .
It's based on jupyter-c-kernel ( actually, it's mostly the same ) and it uses the python implementation of the kernel machinery ( a pure nim version would be cool, but it's outside my abilites at the moment ).
Still, from what I could test, it seems to work rather well as a scratchpad.
Here is an example notebook .
Please note that blocks of code don't share state ( for now at least ) as every block is treated as a different file/executable. I'll try to work on it when I have time (suggestions are welcome).
Also, code completion is not currently implemented, I'll have a look at integrating nimsuggest.
@OderWat: Thanks for the PR!
I didn't think about tempnames, I guess I got lucky ( or I just didn't notice the bug ).
As per disabling hint[Processing], I am on Windows 10 64-bit, and it's still not silent. I guess it's similar to http://forum.nim-lang.org/t/2350/1 ?
I'll try looking into it tomorrow
Let me do an UP on this thread.
@stisa Having a context between the code blocks looks like a non-trivial and very useful task. Did you make any progress? Can I jump in?
A little background: I did some experiments half a year ago. I was able to call a function from Nim. The binary was running, it was looking for a test.nim file to change, then it compiled it to an .so file, dynamically loaded it, and called the function. This was in a loop, so every time I changed the test.nim file, it compiled, loaded, and run it.
@mora: Hi! No I've yet to make any progress, I got sidetracked a bit.
Anyway, feel free to jump in, I'd really love to get a shared context working!
Your solution would still be in python right?
I think your idea is what the original c kernel does, if you look in resources/master.nim there is a ported version commented out, but as I don't understand how it should work the ported code is probably useless.
From what I can understand, it compiles a master as a dynlib that should then execute the actual code the user writes.
There is also a branch here , where I started work on a nim-only kernel, reimplementing messaging etc. but it is still in the early stages ( I think the heartbeat is working and little else ).
Let me now if you make any progress!
I started to play with it. My plan is to keep the python wrapper. The reason is simple: I'm comfortable with python, I don't want to support multiple threads in Nim, and I want to send a kill signal to the process if the user wants to interrupt (menu Kernel->Interrupt in Jupyter) a long running calculation (which signal will be caught in Nim).
I'm planning to have one running Nim binary. The python wrapper will pass the code this binary, which will compile it, and call it with dynlib. The new code will not return, but it will compile the next code, etc. I'm going to use the stack at each run, and I'll not return space. Each new block will have all the previously defined functions and types prepasted before the block's code, and the block's content will be injected within a function, which receives all the previously allocated heap & stack variables. If these variables are var, then it will use the previous layer's stack memory. Furthermore, each block's code will be injected in a block: environment, so you will be able to overwrite previously defined variables with different types. The new local variables will be collected with locals(), so I'll be able to pass them to the next round. In each block the signal used for interrupting the kernel will be overwritten, so upon an interrupt you could continue the work with the latest stack.
Further optimizations: if there is no change in the local variables, then there is no need to keep the stack. For example if the block is x=5 where x is defined in a previous stack, then we don't need to keep this stack. Secondly, if a function is not overloaded (e.g., if it is proc f(x: int):int, then later there is no proc f(x: string):string), then instead of repeating the implementation in the new library, we can simply pass the function as a parameter (I'm just only hope that this will work).
Probably the biggest concern about this proposal is that I'll have a stack overflow if I keep defining new variables in each block. Yes, that's true. I would be still able to load data and play with it interactively. In my Jupyter sessions I rarely reach 100 executed blocks, and I have some optimizations planned.
What do we gain? You could create variables on stack and heap, these won't be (at least not always) copied into the new blocks. You could overwrite previously defined variables, even change their types. It would run with -d:release flag to get speed. And the biggest pro is that I will be able to have something working in days.
@mora: Sounds like a good plan!
I'm still relatively new to nim, so it's probably more detailed than anything I would have thought of.
A couple of questions, sorry if they are obvious / make little sense:
The python wrapper will pass the code this binary, which will compile it, and call it with dynlib. > The new code will not return, but it will compile the next code, etc
So something like like compile master > master compiles block1, runs block1 and adds its variables to master, then master waits for another block to be compiled ?
It would run with -d:release flag to get speed
If I remember correctly, this would disable assert(...) right ?
I hope that I'll be able to create a compiler plugin
Would this require recompiling the compiler or can those be added in a folder or something, maybe while installing (through pypi) ?
Each new block will have all the previously defined functions and types prepasted before the block's code
How do you plan to handle recompilation of a block that defines a type ? I guess you could omit types with the same name?
Secondly, if a function is not overloaded (e.g., if it is proc f(x: int):int, then later there is no proc f(x: string):string), then instead of repeating the > implementation in the new library, we can simply pass the function as a parameter (I'm just only hope that this will work).
Maybe you could convert procs to variables and then shadow them, like:
proc f (x:string) =
echo x
var f = proc (x:string) =
echo x
f("hi") # hi
f = proc (x:string) =
echo "due"&x
f("hi") # duehi
Silvio
Hi Silvio,
My plan is the following: the python code compiles the master, the master compiles and runs block1. Here block1 is modified (we add a bunch of stuff before and after the code) before compiling. And the magic: block1 compiles block2, and runs it, etc.
The -d:release flags do many things. Generally it makes your code faster and less safe (no boundary checks, no assert, etc.)
If we need a compiler plugin, then we will write it for ourselves, then we will try to convince people here to merge it to Nim :)
If I remember correctly, we can override a type within a block. The type resolution does the same lookup technique as the variables.
Hi Peter,
I knew the shadow variable looked too easy! Oh well... :)
( sorry if I keep asking obvious questions, I'm not used to handling memory and the like. I'll have to read up on how memory is handled by nim, the gc, etc )
So you have compile master> master runs block1, forwards its memory/variables/procs etc ( how? ), then exits > block1 waits for block2 and so on?
Mmh I think there's still something I'm missing, as I don't see how you could preserve memory. I don't think just passing a pointer or something would be enough? Is there a way to avoid nim cleaning up memory when you close and executable? ( lol this sounds like intentionally leaking memory )
Unless you left all blocks running, all chained together?
Maybe you could serialize variables with marshal , extending it to keep name and type, and having python hold the serialized data, but this could prove difficult if we ever need to handle large datasets.
About assert, I was just asking as that is something we should document ( and maybe a magic of some kind could be added later on to disable -d:release )
Looks like you're right about Types, this works:
type A = object
x: string
var a = A(x:"hi")
echo a # (x: hi)
block:
var b = A(x:"hi")
echo b # (x: hi)
type A = object
x: int
var c :A = A(x:1)
echo c # (x: 1)
type A = object
x: string
var a = A(x:"hi")
echo a #(x: hi)
proc sum(x,y:A):A=
result = A(x: x.x&y.x)
block:
var b = A(x:"hi")
echo sum(a,b) # (x: hihi)
type A = object
x:string
var c : A = A(x:"ho")
var d : A = A(x:"ho")
echo sum(c,d) #type mismatch
Another question: how would you handle a block being recompiled? For examples, current executing block is block5, user recompiles block1. Would the new block1 just be handled like if it was block6? ( I guess the answer is yes, as I can't think of any way this would break other things )
Thanks, Silvio
ps: sorry, I didn't intend on this reply becoming so long