Mirror of forum.nim-lang.org

10418 :: Testament - Test Failure reFilesDiffer comparing wrong files with one another

• Isofruit (orginal) [2023-08-18T18:51:15+02:00] view original

  I'm currently implementing some compile-time validation for my mapster library and have thus swapped to using testament for my tests in order to cover all possible ways of compiling the project and viewing the output.

  I have a usecase that should not compile when a specific flag is activated. So I set myself up a test that looks like this:

  discard """
    action: "reject"
    matrix: "-d:mapsterValidate; -d:mapsterValidate -d:release"
  """
  import unittest
  
  import mapster
  
  ###### COMPILER FLAG SPECIFIC TEST-SUITES ######
  suite "Testing map - Assignment special cases with no field assignments and validation":
    test """
      1. GIVEN an object type A and B where not every field of B can be mapped to a field on A
      WHEN an instance of A is mapped to an instance of B
      THEN it should crash at compiletime
    """:
      # Given
      type A = object
        str: string
      
      type B = object
        str: string
        num: int
      
      proc mapShouldNotCompile(x: A): B {.map.} = discard # This line will crash compilation via an `error()` call from within the `map` macro.
  Run

  The output I'm getting is weird though, it's talking about file comparisons that failed (?)

  
  FAIL: tests/test_map_validation.nim c -d:mapsterValidate -d:release ( 0.62 sec)
  Test "tests/test_map_validation.nim" in category "pattern"
  Failure: reFilesDiffer
  Expected:
  tests/test_map_validation.nim
  
  Gotten:
  mapster.nim
  
  diff --git a/tmp/diffStrings_a_2rVNVBak b/tmp/diffStrings_b_Qx4pQzH1
  index 44a1117..258bdd5 100644
  --- a/tmp/diffStrings_a_2rVNVBak
  +++ b/tmp/diffStrings_b_Qx4pQzH1
  @@ -1 +1 @@
  -tests/test_map_validation.nim
  \ No newline at end of file
  +mapster.nim
  \ No newline at end of file
  Run

  I'm confused at what's going on here.Why is it comparing my testfile test_map_validation.nim to my project module that I import mapster.nim? I mean, of course those aren't equal, so why is it doing that in the first place?

  What am I doing wrong?

---

• Isofruit (orginal) [2023-08-19T08:18:14+02:00] view original

  After fiddling around with it more I now understand that the even though the issue testament states is "reFilesDiffer" , the actual problem is that it doesn't appear to encounter an error call.

  The reason I emphasize appear is because that's not true. For now I think just being able to see echo statements would help here, as testament eats those and doesn't spit them back out even with --verbose used like this:

  testament --verbose --print run tests/externalAPI/test_map_validation.nim

---

• thindil (orginal) [2023-08-19T09:13:29+02:00] view original

  If you want to import a module from your project, you have to use a full path to it, like import ../../src/mapster. The reason of it is, that the test is a separated program. Thus, import statement knows only the version from your packages' repository.

---

• Isofruit (orginal) [2023-08-19T09:23:01+02:00] view original

  Thanks for the help!

  Tried that out and it does indeed solve the secondary issue that was introduced by moving the test into a subdirectory.

  Sadly, the underlying issue - namely that somehow testament receives mapster.nim when it expects tests/externalAPI/test_map_validation.nim - is going strong. Searching the docs doesn't help to explain here where it is getting those values from either. Maybe testament is just not good at dealing with macros, similar to compiles? I'm grasping at straws here.

  
  FAIL: tests/externalAPI/test_map_validation.nim c --define:mapsterValidate ( 1.90 sec)
  Test "tests/externalAPI/test_map_validation.nim" in category "externalAPI"
  Failure: reFilesDiffer
  Expected:
  tests/externalAPI/test_map_validation.nim
  
  Gotten:
  mapster.nim
  Run

---

• Isofruit (orginal) [2023-08-19T13:08:25+02:00] view original

  At this point I believe this to be a bug in testament, as a much simpler example also fails that based on anything I can see documented should work:

  # tplay.nim
  
  discard """
    action: "reject"
  """
  import ./breakPragma
  
  proc x() {.justDo.} = discard
  
  # breakPragma.nim
  
  import std/macros
  
  macro justDo*(procDef: typed): untyped =
    error("I break")
    return procDef
  Run

  I filed an issue on github accordingly: https://github.com/nim-lang/Nim/issues/22514

---