Hello,
I've done some testing with Nimpy and Arraymancer and I've run into a bit of an issue when trying to use Numpy function (via Nimpy) and Arraymancer because several functions have the same name.
For a example, I can't use a numpy.reshape on a PyObject because it conflicts with the reshape functions of arraymancer.
Is there any way work around this or do I have to create a python package that wraps and rename all the python function I want to use ?
Second (unrelated) question, how to create a numpy array (as a PyObject) from a multidimensionnal (3D or above) Tensor ? I've looked into https://github.com/zaferarican/nimpy_numpy_examples/blob/master/nimpy_numpy_examples/nimpy_integral_arraymancer.nim to do the opposite (multidimensionnal tensor from PyObject numpy array).
you can use numpy.reshape(ndarray, shape) and arraymancer.reshape(tensor, shape). The module name can serve as namespacing prefix when calling functions.
Also your object types PyObject vs Tensor should disambiguate what to call for the compiler so I would be surprised if there was conflict, do you have an example to reproduce?
I personally didn't look into Numpy interop yet. My plan was to introduce zero-copy interop with Numpy once this PR lands https://github.com/mratsim/Arraymancer/pull/420 but it's being blocked by a Nim bug: https://github.com/nim-lang/Nim/issues/13193
That said, Arraymancer supports both read and write from .npy files, see tests: https://github.com/mratsim/Arraymancer/blob/28a0a255/tests/io/test_numpy.nim. The only limitation is that you need to pass the data type.
let a = read_npy[int64](filePathIn)
a.write_npy(filePathOut)
The following code gives me a compile time error :
import arraymancer
import nimpy
import nimpy/raw_buffers
proc `+`[T](p: ptr T, val: int) : ptr T {.inline.}=
cast[ptr T](cast[uint](p) + cast[uint](val * sizeof(T)))
proc pyBufToTensor[T](ndArray: PyObject): Tensor[T]=
# Convert PyObject to RawPyBuffer
var aBuf: RawPyBuffer
ndArray.getBuffer(aBuf, PyBUF_WRITABLE or PyBUF_ND)
aBuf.release()
# Copy buffer into Tensor
var shape: seq[int]
for i in 0..<aBuf.ndim:
shape.add((aBuf.shape+i)[])
# Get memory asdress of buffer
var bufPtr = cast[ptr UncheckedArray[T]](aBuf.buf)
# Get underlying data buffer of Tensor
result = newTensor[T](shape)
var tensorDataPtr = cast[ptr UncheckedArray[T]](result.get_data_ptr)
# Copy Buffer into Tensor
var length = shape.foldl(a*b)
copyMem(tensorDataPtr, bufPtr, length*sizeof(T))
let np = pyImport("numpy")
proc nimfftshift[T](t: Tensor[T]): Tensor[T]=
## Reshape PyObject to Arraymancer Tensor
#var ndArray = wrappy.w_fftshift(t.toSeq, t.shape.toSeq)
var shape = np.array(t.shape.toSeq)
var ndArray = np.array(t.toSeq)
ndArray = np.reshape(ndArray, shape)
ndArray = np.fft.fftshift(ndArray)
# Convert RawPyBuffer to Tensor
result = pyBufToTensor[T](ndArray)
but expected one of:
proc reshape(t: Tensor; new_shape: MetadataArray): Tensor
first type mismatch at position: 1
required type for t: Tensor
but expression 'np' is of type: PyObject
proc reshape(t: Tensor; new_shape: varargs[int]): Tensor
first type mismatch at position: 1
required type for t: Tensor
but expression 'np' is of type: PyObject
proc reshape[TT](a: Variable[TT]; shape: MetadataArray): Variable[TT]
first type mismatch at position: 1
required type for a: Variable[reshape.TT]
but expression 'np' is of type: PyObject
proc reshape[TT](a: Variable[TT]; shape: varargs[int]): Variable[TT]
first type mismatch at position: 1
required type for a: Variable[reshape.TT]
but expression 'np' is of type: PyObject
expression: reshape(np, ndArray, shape)
My plan was to introduce zero-copy interop with Numpy
That would be awesome !
After further investigation it's not a name conflict issue.
This compiles
import ../src/arraymancer
import nimpy
import nimpy/raw_buffers
import sequtils
proc `+`[T](p: ptr T, val: int) : ptr T {.inline.}=
cast[ptr T](cast[uint](p) + cast[uint](val * sizeof(T)))
proc pyBufToTensor[T](ndArray: PyObject): Tensor[T]=
# Convert PyObject to RawPyBuffer
var aBuf: RawPyBuffer
ndArray.getBuffer(aBuf, PyBUF_WRITABLE or PyBUF_ND)
aBuf.release()
# Copy buffer into Tensor
var shape: seq[int]
for i in 0..<aBuf.ndim:
shape.add((aBuf.shape+i)[])
# Get memory asdress of buffer
var bufPtr = cast[ptr UncheckedArray[T]](aBuf.buf)
# Get underlying data buffer of Tensor
result = newTensor[T](shape)
var tensorDataPtr = cast[ptr UncheckedArray[T]](result.get_data_ptr)
# Copy Buffer into Tensor
var length = shape.foldl(a*b)
copyMem(tensorDataPtr, bufPtr, length*sizeof(T))
let np = pyImport("numpy")
proc nimfftshift(t: Tensor[float32]): Tensor[float32]=
## Reshape PyObject to Arraymancer Tensor
#var ndArray = wrappy.w_fftshift(t.toSeq, t.shape.toSeq)
var shape = np.`array`(t.shape.toSeq)
var ndArray = np.`array`(t.toSeq)
ndArray = np.reshape(ndArray, shape)
ndArray = np.fft.fftshift(ndArray)
# Convert RawPyBuffer to Tensor
result = pyBufToTensor[float32](ndArray)
proc main() =
let a = randomTensor([3, 3], 3).asType(float32)
echo a
let b = nimfftshift(a)
echo b
main()
The difference is that pyBufToTensor is not a generic proc.
The underlying reason is a very thorny problem in Nim. In generic proc, Nim tries to resolve symbols and statement early, including untyped symbol. But nimpy relies on untyped symbol to do its magic: https://github.com/yglukhov/nimpy/blob/b5401247
template `.()`*(o: PyObject, field: untyped, args: varargs[untyped]): PyObject =
dotCall(o, field, args)
template `.`*(o: PyObject, field: untyped): PyObject =
getAttr(o, astToStr(field))
So it does not have the opportunity to transform np.reshape into a wrapped call before Nim compiler tries to check if the reshape symbol is valid in generic procs.
Unfortunately that is a bug that is very hard to solve.
That would be awesome ! Is there a way to call numpy API directly in arraymancer or do you still use Nimpy ?
No there is nothing for now.
Uh wouldn't have guessed it would be such a complex issue actually.
Well, thanks for the heads up, i'll just use non-generic proc.
Unfortunately that is a bug that is very hard to solve
Ok, but the workaround is trivial, no? Just use nimpy.callMethod directly instead of the .() magic.