[SR-1124] Drop FileCheck from test/1_stdlib, use StdlibUnittest instead

[modocache]

Previous ID	SR-1124
Radar	None
Original Reporter	@modocache
Type	Improvement

Additional Detail from JIRA

Votes	0
Component/s	Standard Library
Labels	Improvement, StarterBug, Test
Assignee	None
Priority	Medium

md5: 5245d992633498d3acacd3655548dd7c

Issue Description:

Swift's test/1_stdlib directory is meant to include tests for the Swift stdlib. Most of these are written using the testing framework designed for this purpose, StdlibUnittest. However, some use an ad-hoc system of printing strings to stdout, then checking the output via FileCheck.

The following tests should be written using StdlibUnittest (and no longer reference FileCheck):

1. test/1_stdlib/ArrayBridge.swift

2. test/1_stdlib/BitwiseOperationsType.swift

3. test/1_stdlib/BridgeNonVerbatim.swift

4. test/1_stdlib/Bridgeable.swift

5. test/1_stdlib/CGGeometry.swift

6. test/1_stdlib/Collection.swift

7. test/1_stdlib/CollectionOfOne.swift

8. test/1_stdlib/Float.swift

9. test/1_stdlib/FloatingPointIR.swift

10. test/1_stdlib/HeapBuffer.swift

11. test/1_stdlib/ImplicitlyUnwrappedOptional.swift

12. test/1_stdlib/KVO.swift

13. test/1_stdlib/LogicValue.swift

14. test/1_stdlib/Map.swift

15. test/1_stdlib/NSObject.swift

16. test/1_stdlib/NSUndoManager.swift

17. test/1_stdlib/NewArray.swift.gyb

18. test/1_stdlib/NewString.swift

19. test/1_stdlib/NewStringAppending.swift

20. test/1_stdlib/Nil.swift

21. test/1_stdlib/Reflection.swift

22. test/1_stdlib/Reflection_jit.swift

23. test/1_stdlib/Reflection_objc.swift

24. test/1_stdlib/SegmentAlignment.c

25. test/1_stdlib/SpriteKit.swift

26. test/1_stdlib/StringReallocation.swift

27. test/1_stdlib/TypeName.swift

28. test/1_stdlib/UIKit.swift

29. test/1_stdlib/UIViewControllerAdditions.swift

30. test/1_stdlib/VarArgs.swift

31. test/1_stdlib/Zip.swift

32. test/1_stdlib/alloc_rounding.swift

33. test/1_stdlib/integer_conversions.swift

34. test/1_stdlib/sort_integers.swift

35. test/1_stdlib/subString.swift

[belkadan]

I'm a little concerned that we'll lose what little coverage we have for the interpreter; right now there's a (non-buildbot) test mode that turns simple compile-and-run invocations into interpreter invocations. StdlibUnittest currently doesn't support that because it runs all tests out of process by default.

I should probably file a bug for this.

[belkadan]

Filed SR-1125 and SR-1126 (neither of which should block this effort).

[swift-ci]

Comment by Kyle Yoon (JIRA)

Is this task mostly changing

// RUN: %target-run %t/ArrayBridge > %t.txt
// RUN: FileCheck %s < %t.txt

into

// RUN: %target-run-stdlib-swift | FileCheck %s

for the listed files?
Also, does it require you to re-build the toolchain and test every time?

Sorry for all the questions, again!

[modocache]

No need to apologize for clarifying the task! On the contrary, thanks for these follow-up questions.

The ideal end state of the listed tests would be the following:

// RUN: %target-run-simple-swift

A good example is test/1_stdlib/NSEnumeratorAPI.swift. Notice that that test does not use FileCheck at all. Instead, it uses the TestSuite construct from StdlibUnittest. This library is capable of making assertions that are not based on textual output, like FileCheck is. We want to avoid text output-based assertions.

[swift-ci]

Comment by Sammy Gutierrez (JIRA)

Here is the work I have for this ticket.

I was hoping someone could take a quick look over it before submitting the PR (should I ask for help in the forums instead?). I attempted to remove FileCheck from any files that I could in /test/stdlib but there were a few places where I did not know what to do:

• Reflection_jit.swift - Could not get Reflection.swift tests to run with JIT (deleted this file for now)

• SwiftObjectNSObject.swift - Calls SwiftObjectNSObject.m and checks for output.

[belkadan]

cc @airspeedswift

[belkadan]

Resetting assignee for all Starter Bugs not modified since 2018.

I think we lost this one since it wasn't a full PR. In the future, please feel free to make a PR even if it's still WIP.