Mirror of forum.nim-lang.org

9393 :: Nimble and two different C files with the same name

• radekm (orginal) [2022-08-19T22:15:35+02:00] view original

  Hi,

  I'm trying to wrap libdeflate in Nim. I would like to create a package which doesn't depend on any external library so I embedded C sources of libdeflate inside my Nimble package.

  Now I use following compile directives in my source code:

  {.compile: "libdeflate_gzip/private/libdeflate/lib/deflate_decompress.c",
    compile: "libdeflate_gzip/private/libdeflate/lib/utils.c",
    compile: "libdeflate_gzip/private/libdeflate/lib/arm/cpu_features.c",
    compile: "libdeflate_gzip/private/libdeflate/lib/x86/cpu_features.c",
    compile: "libdeflate_gzip/private/libdeflate/lib/crc32.c",
    compile: "libdeflate_gzip/private/libdeflate/lib/gzip_decompress.c"
  .}
  Run

  Unfortunately when linking my test I'm getting error

  
  duplicate symbol '_libdeflate_init_x86_cpu_features' in:
      /Users/radekm/.cache/nim/test1_d/cpu_features.c.o
  Run

  The problem is that there are two files cpu_features.c with the same name but in different directories. Then my linker is called with

  
  /Users/radekm/.cache/nim/test1_d/cpu_features.c.o
  Run

  twice and I have duplicate symbols. I also suspect that cpu_features.c for x86 ovewrites one for ARM. Can you advice how can I solve this without modifying or renaming the original sources of libdeflate?

---

• treeform (orginal) [2022-08-20T03:37:43+02:00] view original

  I don't know how to solve your main problem, it feels like a c compiler bug?

  But https://github.com/guzba/zippy implement gzip decompression in native nim and is very fast, beating or matching zlib. It also uses different SIMD for x86 and arm. Maybe no need to wrap libdeflate?

---

• radekm (orginal) [2022-08-20T08:45:51+02:00] view original

  Thanks for the answer. The problem with Zippy is that it doesn't support gzips with multiple members ie. multiple concatenated gzips (which is according to the standard RFC 1952 still valid gzip and all unix tools normally support that).

  The culprit in Zippy is https://github.com/guzba/zippy/blob/master/src/zippy/gzip.nim#L64 where it's reading size and CRC from the end of the whole file. But this doesn't work if we concatenate a.gz with b.gz because then at the end of the file is size and CRC from b.gz but zippy tries to use it to verify a.gz.

  So the result is

  
  Error: unhandled exception: Checksum verification failed [ZippyError]
  Run

---

• guibar (orginal) [2022-08-20T09:06:07+02:00] view original

  Maybe with conditional compilation on arm :

  {.compile: "libdeflate_gzip/private/libdeflate/lib/deflate_decompress.c",
    compile: "libdeflate_gzip/private/libdeflate/lib/utils.c",
    compile: "libdeflate_gzip/private/libdeflate/lib/crc32.c",
    compile: "libdeflate_gzip/private/libdeflate/lib/gzip_decompress.c"
  .}
  
  when defined(arm) or defined(arm64):
    {.compile: "libdeflate_gzip/private/libdeflate/lib/arm/cpu_features.c".}
  else:
    {.compile: "libdeflate_gzip/private/libdeflate/lib/x86/cpu_features.c".}
  
  Run

---

• radekm (orginal) [2022-08-20T09:21:45+02:00] view original

  Awesome. You're right, it works!

---

• mratsim (orginal) [2022-08-20T16:19:44+02:00] view original

  You can probably rename the C file on compilation, see: https://github.com/numforge/agent-smith/blob/a2d9251e289f92f6b5fb68e19a98d16b00f2694c/third_party/ale_build.nim#L79-L82

  {.compile: "libdeflate_gzip/private/libdeflate/lib/deflate_decompress.c",
    compile: "libdeflate_gzip/private/libdeflate/lib/utils.c",
    compile: ("libdeflate_gzip/private/libdeflate/lib/arm/cpu_features.c", "cpu_features_arm.c.o"),
    compile: ("libdeflate_gzip/private/libdeflate/lib/x86/cpu_features.c", "cpu_features_x86.c.o"),
    compile: "libdeflate_gzip/private/libdeflate/lib/crc32.c",
    compile: "libdeflate_gzip/private/libdeflate/lib/gzip_decompress.c"
  .}
  Run

---