[Mono.Android] API-26 Enumification (dotnet#662)

Fixes: https://bugzilla.xamarin.com/show_bug.cgi?id=30909
Fixes: https://bugzilla.xamarin.com/show_bug.cgi?id=51293
Fixes: https://bugzilla.xamarin.com/show_bug.cgi?id=55473

Bump to xamarin-android-api-compatibility/master/6fe9b17f.

Add `Documentation/HowToAddNewApiLevel.md`, to describe the process
for binding new API levels, including enumification.

Enumify API-26.

Update `build-tools/enumification-helpers` to support the latest
droiddoc `packages.html` format.

API-26 changed `Activity.findViewById(int)` to be a generic method.
Remove the `<typeParameters/>` element and "fix" the return type so
that we don't break API compatibility, as `generator` doesn't
currently do anything sane here. (Specifically, all generic
paramaters are currently bound as `Java.Lang.Object`, *even if* the
type parameter has a constraint which provides a more specific type!)

Fix binding of `Java.Nio.FileChannel` so it can be actually bound,
related `SeekableByteChannel` support fixes.

Remove `//implements[@name='AutoCloseable']` and
`//implements[@name='Destryable']`, as they add Java `default`
interface methods which can't be sanely supported at this time.

Update `api-merge` to change `default` interface methods to
non-`default` methods when merging with an older API level which
contained the non-`default` interface method. This preserves backward
compatibility with existing assemblies and source code. Remove the
associated workarounds.

Bump xamarin-android-api-compatibility to check against the
latest-and-greatest API-26 binding.

Documentation/HowToAddNewApiLevel.md

@@ -0,0 +1,32 @@
+
+# How to deal with new API Level
+
+## This documentation is incomplete
+
+In Xamarin ages, we used to have (more complete) API upgrade guide internally. But since then we switched to new xamarin-android repository which entirely changed the build system from Makefile to MSBuild solution, as well as the managed API for manipulating Android SDK, the old documentation almost does not make sense anymore. Even though I am writing this documentation, I don't know everything required (nor those who changed the build system didn't care about API upgrades).
+
+Hence, this documentation is written from the ground,  exploring everything around.
+
+And since the build system has changed between the first preview of Android O and the latest Android O preview (3), and it is quite possible that the build system changes over and over again, it might still not make much sense in the future.
+
+## Steps
+
+Anyhow, this commit would tell you what needs to be changed when the new API preview arrives (and that becomes stable): https://github.com/xamarin/xamarin-android/pull/642
+
+1) Add/update new download to build-tools/android-toolchain.
+The new API archive should be found on the web repository description
+that Google Android SDK Manager uses (which can be now shown as part
+of SDK Manager options in Android Studio).
+
+As of Android O ages, it could be found at https://dl-ssl.google.com/android/repository/repository2-1.xml . It used to be different repository description URL, and it will be different URL in the future.
+
+2) Create and add api-O.xml.in under src/Mono.Android/Profile directory.
+It can be done from within `build-tools/api-xml-adjuster` directory. See `README.md` in that directory for details. You will have to make changes to Makefile in that directory to generate it. Also note that it often happens that Google does not ship documentation archive (!) and in such case we will have to scrape DroidDoc from the web and create our own docs archive.
+
+3) Review new API (or review the changes in case of API updates).
+
+Some of the changes are caused by API removal. Usually they come with
+"deprecated" state in the previous API Level first, but that's not always true.
+
+4) enumification: See `build-tools/enumification-helpers/README`.
+

build-tools/api-merge/ApiDescription.cs

@@ -121,14 +121,21 @@ public void Merge (string apiLocation)
 								var sa = smember.Attribute (a.Name);
 								if (sa == null)
 									smember.Add (a);
-								else if (sa.Value != a.Value)
-									sa.SetValue (a.Value);
-							}
+								else {
+									// An interface method can become a default interface method in
+									// the later API Levels, but since such a method still has to be 
+									// explicitly implemented in older API versions, they should be non-default.
+									// So, do not change this attribute if it is an interface method.
+									if (sa.Name.LocalName == "abstract") {
 #if KEEP_OLD_WRONG_COMPATIBILITY
-							var isAbstract = smember.Attribute ("abstract");
-							if (isAbstract != null)
-								isAbstract.Value = nmember.Attribute ("abstract").Value;
+										if (stype.Name.LocalName != "interface")
+											sa.Value = a.Value;
 #endif
+									}
+									else if (sa.Value != a.Value)
+										sa.SetValue (a.Value);
+								}
+							}
 							// This is required to workaround the
 							// issue that API Level 20 does not offer
 							// reference docs and we use API Level 21

build-tools/enumification-helpers/.gitignore

@@ -0,0 +1,7 @@
+map.ext.csv
+const-list-*.xml
+remaining-int-methods.txt
+*.exe
+*.mdb
+*.pdb
+

build-tools/enumification-helpers/Makefile

@@ -1,11 +1,12 @@
 topdir = ../..
 
-API_LEVEL = 25
-API_LEVEL_NAME = 25
+API_LEVEL = 26
+API_LEVEL_NAME = 26
+
+CSCOMPILE = csc -debug:portable
 
 CONFIGURATION = Debug
-API_XML_ANALYZER_DIR = $(topdir)/build-tools/api-xml-adjuster
-JAR2XML_DIR = $(topdir)/external/jar2xml
+DOCS_DIR=~/android-toolchain/docs/docs-api-$(API_LEVEL_NAME)
 
 CONST_MAPPER_SOURCES = \
 	generate-const-mapping.cs \
@@ -17,20 +18,20 @@ all:: map.ext.csv remaining-int-consts.txt remaining-int-methods-filtered.txt
 #enum mappings
 
 generate-const-list.exe: generate-const-list.cs
-	mcs generate-const-list.cs -debug
+	$(CSCOMPILE) generate-const-list.cs
 
 const-list-$(API_LEVEL).xml: generate-const-list.exe
-	mono --debug generate-const-list.exe $(API_XML_ANALYZER_DIR)/docs-api-$(API_LEVEL_NAME)/reference/ -v > const-list-$(API_LEVEL).xml || rm const-list-$(API_LEVEL).xml
+	mono --debug generate-const-list.exe $(DOCS_DIR)/reference/ -v > const-list-$(API_LEVEL).xml || rm const-list-$(API_LEVEL).xml
 
 generate-intermediary-enum-candidates.exe : generate-intermediary-enum-candidates.cs
-	mcs generate-intermediary-enum-candidates.cs -debug
+	$(CSCOMPILE) generate-intermediary-enum-candidates.cs
 
 #it is obsolete
 intermediary-enum-candidates.xml: generate-intermediary-enum-candidates.exe
 	mono --debug generate-intermediary-enum-candidates.exe $(topdir)/src/Mono.Android/api-$(API_LEVEL).xml.in > intermediary-enum-candidates.xml || rm intermediary-enum-candidates.xml
 
 generate-const-mapping.exe: $(CONST_MAPPER_SOURCES)
-	mcs $(CONST_MAPPER_SOURCES) -debug
+	$(CSCOMPILE) $(CONST_MAPPER_SOURCES)
 
 map.ext.csv: generate-const-mapping.exe enum-conversion-mappings.xml const-list-$(API_LEVEL).xml
 	mono --debug generate-const-mapping.exe enum-conversion-mappings.xml const-list-$(API_LEVEL).xml > map.ext.csv || rm map.ext.csv
@@ -48,22 +49,22 @@ remaining-int-consts.txt: ../../src/Mono.Android/obj/$(CONFIGURATION)/android-$(
 
 # method mappings
 generate-intermediary-doc-enum-mapping.exe: generate-intermediary-doc-enum-mapping.cs
-	mcs generate-intermediary-doc-enum-mapping.cs -debug
+	$(CSCOMPILE) generate-intermediary-doc-enum-mapping.cs
 
 generate-enumlist-to-query.exe : generate-enumlist-to-query.cs
-	mcs generate-enumlist-to-query.cs -debug
+	$(CSCOMPILE) generate-enumlist-to-query.cs
 
 generate-intermediary-method-mapping.exe: generate-intermediary-method-mapping.cs
-	mcs generate-intermediary-method-mapping.cs -debug
+	$(CSCOMPILE) generate-intermediary-method-mapping.cs
 
 intermediary-enum-list.txt: generate-enumlist-to-query.exe
 	mono --debug generate-enumlist-to-query.exe const-list-$(API_LEVEL).xml > intermediary-enum-list.txt || rm intermediary-enum-list.txt
 
 intermediary-doc-enum-mapping.tsv: generate-intermediary-doc-enum-mapping.exe intermediary-enum-list.txt
-	mono --debug generate-intermediary-doc-enum-mapping.exe $(API_XML_ANALYZER_DIR)/docs-api-$(API_LEVEL_NAME)/reference intermediary-enum-list.txt > intermediary-doc-enum-mapping.tsv || rm intermediary-doc-enum-mapping.tsv
+	mono --debug generate-intermediary-doc-enum-mapping.exe $(DOCS_DIR)/reference intermediary-enum-list.txt > intermediary-doc-enum-mapping.tsv || rm intermediary-doc-enum-mapping.tsv
 
 intermediary-method-mapping.txt: generate-intermediary-method-mapping.exe
-	mono --debug generate-intermediary-method-mapping.exe intermediary-doc-enum-mapping.tsv const-list-$(API_LEVEL).xml $(API_XML_ANALYZER_DIR)/docs-api-$(API_LEVEL_NAME)/reference > intermediary-method-mapping.txt || rm intermediary-method-mapping.txt
+	mono --debug generate-intermediary-method-mapping.exe intermediary-doc-enum-mapping.tsv const-list-$(API_LEVEL).xml $(DOCS_DIR)/reference > intermediary-method-mapping.txt || rm intermediary-method-mapping.txt
 
 intermediary-field-candidates.txt: intermediary-method-mapping.txt
 	grep "#[a-z]" intermediary-method-mapping.txt  | grep -v '('  > intermediary-field-candidates.txt

build-tools/enumification-helpers/README

@@ -193,6 +193,16 @@ The results are to be saved as "map.ext.csv" and "methodmap.ext.csv" that are to
 	    after "// automatically generated ones" line) just edit map.csv.
 	  - then run "make API_LEVELS=[latest] clean all" in src/Mono.Android
 	    to build enumified dll.
+	  - note tht API Level description in the API docs are used to
+	    become part of the CSV entry. The first item is the API Level.
+	    That often brings problem. This is what happened when API Level
+	    26 as stable has arrived: Google had published "android-26" API
+	    which used to be "android-O", they still hadn't published the
+	    corresponding docs and therefore the docs say "O" but the actual
+	    API was "26". That later brought confusion at generator.exe ran.
+	    To workaround this issue, replace the wrong API level specifier
+	    (e.g. "O") with the actual API level (e.g. "26"). Use regex like
+	    /^O/26/ to achieve this.
 	  - go back to this directory and rebuild remaining-int-consts.txt
 	    and check if the enumified lines disappeared.
 	  - There are some constaints that are NOT enumified
@@ -221,6 +231,15 @@ The results are to be saved as "map.ext.csv" and "methodmap.ext.csv" that are to
 		  convert to enums" methods, or blacklist-field.txt for fields.
 		- in general those notes do not describe *everything*. git diff
 		  would tell what is new and what should be taken care this time.
+	  - Here is how I worked on it at API Level 26 to add new lines to methodmap.ext.csv:
+		- Read git diff on remaining-int-consts.txt to iterate all the new enum types.
+		- For each new enum type that (typically) has prefix P_ in type T, I run `grep -R P_ | grep T | grep "\w*.html:"` in the local reference docs, to see which documentation HTML mention that const (I ran the command on GNOME terminal, so I got the final grep matches with color highlights).
+		- And for each HTML doc, look up that const and add records to methodmap.ext.csv.
+		- I usually bring in mistakes. When there are wrong const mappings, they will result in binding generator warnings for unmatched XPath selectors that you can find them by looking up enummetadata.
+		- For method mappings, mistakes can be twofolds:
+		  - For the wrong matching, you'll see them as binding generator warnings for unmatched XPath selectors just like const mappings.
+		  - For the wrong replacement enum type, that's NOT going to be unmatched XPath. Instead it will result in a reference to missing type warning that can be still found as a generator warning (but harder to find).
+		  - It is often hard to identify why the mappings are regarded as invalid; sometimes Google API reference tells you lies(!) that there are some methods that actually don't exist(!). You'd like to check `obj/Debug/android-*/api.xml`, `obj/Debug/android-*/api.xml.fixed` (XML after applying metadata fixup) or `Profiles/api-*.xml.in` (for "raw" API data before `api-merge` step).
 	  - At some stage, go to src/Mono.Android and append
 	    methodmap.ext.csv contents to methodmap.csv and run
 	    "make API_LEVELS=[latest] clean all" to get enumified dll.