Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Failing to get proper visual observation on a Linux build using HDRP #3835

Closed
automata opened this issue Apr 23, 2020 · 11 comments
Closed

Failing to get proper visual observation on a Linux build using HDRP #3835

automata opened this issue Apr 23, 2020 · 11 comments
Assignees
@vincentpierre
Labels
bug Issue describes a potential bug in ml-agents.

Comments

@automata
Copy link
Contributor

Describe the bug
While using the HDRP pipeline, we're able to create and train following the steps from https://github.com/Unity-Technologies/ml-agents/blob/latest_release/docs/Learning-Environment-Create-New.md.

We can even run a Jupyter notebook, send actions and get both vector and visual observations from the running environment inside Unity Editor.

However, when we build for Linux and try to call the binary from the Jupyter notebook, we're able to run actions but resulting visual observation is the right camera shot the first time but just dark images (with some incomplete representations as next results), as you can see in the following screenshot.

We tested the same project on Windows 10 and it works, from inside Unity Editor or a built binary for Windows.

To Reproduce
Steps to reproduce the behavior:

  1. Create an Unity project using HDRP
  2. Follow the steps on https://github.com/Unity-Technologies/ml-agents/blob/latest_release/docs/Learning-Environment-Create-New.md to create a new learning environment
  3. Capture visual observations from the main camera
  4. Build a Linux binary
  5. Run the Jupyter notebook from https://github.com/Unity-Technologies/ml-agents/blob/latest_release/notebooks/getting-started.ipynb but using the binary you just built
  6. Check the visual observations

Console logs / stack traces
We don't see any error or other relevant information on console logs.

Screenshots
Screenshot from 2020-04-23 17-33-32

Environment:

  • OS + version: Ubuntu Linux 18.04
  • ML-Agents version: 0.15.1
  • TensorFlow version: 1.14.0
  • Python: 3.6.9
  • Environment: RollerAgent (built from docs in a new HDRP project)
@automata automata added the bug Issue describes a potential bug in ml-agents. label Apr 23, 2020
@vincentpierre vincentpierre self-assigned this Apr 23, 2020
@vincentpierre
Copy link
Contributor

vincentpierre commented Apr 23, 2020 •
edited
Loading

Hi,

What are the graphics settings you have on Linux ? (Are you using Vulkan ?)
Are you able to run the environment on Linux without issues ?
Are you running in batchmode or with the no-graphics command line ?
Do you have some Player logs that could help us debug this?

@automata
Copy link
Contributor Author

Hey!

Yes, I'm using Vulkan, on a GTX 1060 GPU.

Yes, I can run the environment with zero issues on Linux.

I tried to run it just by calling ./hdrp_cam.x86_64 -force-vulkan --mlagents-port 5004 and then running the Python code with UnityEnvironment(file_name=None, side_channels = [engine_configuration_channel], base_port=5004). Also tried running specifying the file_name. Both get me the same issue. Only runs flawlessly when inside Unity Editor ((2019.3.10f1).

The Player logs was a good call! Here you go:

Mono path[0] = '/media/data/tmp/unity/hdrp_cam_Data/Managed'
Mono config path = '/media/data/tmp/unity/hdrp_cam_Data/MonoBleedingEdge/etc'
PlayerConnection initialized from /media/data/tmp/unity/hdrp_cam_Data (debug = 0)
PlayerConnection initialized network socket : 0.0.0.0 55495
Multi-casting "[IP] 192.168.0.10 [Port] 55495 [Flags] 2 [Guid] 1673654108 [EditorId] 0 [Version] 1048832 [Id] LinuxPlayer(192.168.0.10) [Debug] 0 [PackageName] LinuxPlayer [ProjectName] <no name>" to [225.0.0.222:54997]...
Started listening to [0.0.0.0:55495]
Preloaded 'lib_burst_generated.so'
Preloaded 'libgrpc_csharp_ext.x64.so'
PlayerConnection already initialized - listening to [0.0.0.0:55495]
Display 0 'LG ULTRAWIDE 29"': 2560x1080 (primary device).
Desktop is 2560 x 1080 @ 60 Hz
Invalid initial resolution 80 x 80 - forcing to 100 x 100
[Vulkan init] extensions: count=18
[Vulkan init] extensions: name=VK_KHR_device_group_creation, enabled=0
[Vulkan init] extensions: name=VK_KHR_external_fence_capabilities, enabled=0
[Vulkan init] extensions: name=VK_KHR_external_memory_capabilities, enabled=0
[Vulkan init] extensions: name=VK_KHR_external_semaphore_capabilities, enabled=0
[Vulkan init] extensions: name=VK_KHR_get_display_properties2, enabled=0
[Vulkan init] extensions: name=VK_KHR_get_physical_device_properties2, enabled=0
[Vulkan init] extensions: name=VK_KHR_get_surface_capabilities2, enabled=0
[Vulkan init] extensions: name=VK_KHR_surface, enabled=1
[Vulkan init] extensions: name=VK_KHR_surface_protected_capabilities, enabled=0
[Vulkan init] extensions: name=VK_KHR_wayland_surface, enabled=1
[Vulkan init] extensions: name=VK_KHR_xcb_surface, enabled=0
[Vulkan init] extensions: name=VK_KHR_xlib_surface, enabled=1
[Vulkan init] extensions: name=VK_KHR_display, enabled=1
[Vulkan init] extensions: name=VK_EXT_direct_mode_display, enabled=0
[Vulkan init] extensions: name=VK_EXT_acquire_xlib_display, enabled=0
[Vulkan init] extensions: name=VK_EXT_display_surface_counter, enabled=0
[Vulkan init] extensions: name=VK_EXT_debug_report, enabled=1
[Vulkan init] extensions: name=VK_EXT_debug_utils, enabled=0
Vulkan detection: 2
Initialize engine version: 2019.3.10f1 (5968d7f82152)
[Subsystems] Discovering subsystems at path /media/data/tmp/unity/hdrp_cam_Data/UnitySubsystems
Forcing GfxDevice: Vulkan
GfxDevice: creating device client; threaded=1
[Vulkan init] extensions: count=18
[Vulkan init] extensions: name=VK_KHR_device_group_creation, enabled=0
[Vulkan init] extensions: name=VK_KHR_external_fence_capabilities, enabled=0
[Vulkan init] extensions: name=VK_KHR_external_memory_capabilities, enabled=0
[Vulkan init] extensions: name=VK_KHR_external_semaphore_capabilities, enabled=0
[Vulkan init] extensions: name=VK_KHR_get_display_properties2, enabled=0
[Vulkan init] extensions: name=VK_KHR_get_physical_device_properties2, enabled=0
[Vulkan init] extensions: name=VK_KHR_get_surface_capabilities2, enabled=0
[Vulkan init] extensions: name=VK_KHR_surface, enabled=1
[Vulkan init] extensions: name=VK_KHR_surface_protected_capabilities, enabled=0
[Vulkan init] extensions: name=VK_KHR_wayland_surface, enabled=1
[Vulkan init] extensions: name=VK_KHR_xcb_surface, enabled=0
[Vulkan init] extensions: name=VK_KHR_xlib_surface, enabled=1
[Vulkan init] extensions: name=VK_KHR_display, enabled=1
[Vulkan init] extensions: name=VK_EXT_direct_mode_display, enabled=0
[Vulkan init] extensions: name=VK_EXT_acquire_xlib_display, enabled=0
[Vulkan init] extensions: name=VK_EXT_display_surface_counter, enabled=0
[Vulkan init] extensions: name=VK_EXT_debug_report, enabled=1
[Vulkan init] extensions: name=VK_EXT_debug_utils, enabled=0
[Vulkan init] extensions: count=89
[Vulkan init] extensions: name=VK_KHR_8bit_storage, enabled=0
[Vulkan init] extensions: name=VK_KHR_16bit_storage, enabled=0
[Vulkan init] extensions: name=VK_KHR_bind_memory2, enabled=0
[Vulkan init] extensions: name=VK_KHR_create_renderpass2, enabled=0
[Vulkan init] extensions: name=VK_KHR_dedicated_allocation, enabled=1
[Vulkan init] extensions: name=VK_KHR_depth_stencil_resolve, enabled=0
[Vulkan init] extensions: name=VK_KHR_descriptor_update_template, enabled=1
[Vulkan init] extensions: name=VK_KHR_device_group, enabled=0
[Vulkan init] extensions: name=VK_KHR_draw_indirect_count, enabled=0
[Vulkan init] extensions: name=VK_KHR_driver_properties, enabled=0
[Vulkan init] extensions: name=VK_KHR_external_fence, enabled=0
[Vulkan init] extensions: name=VK_KHR_external_fence_fd, enabled=0
[Vulkan init] extensions: name=VK_KHR_external_memory, enabled=0
[Vulkan init] extensions: name=VK_KHR_external_memory_fd, enabled=0
[Vulkan init] extensions: name=VK_KHR_external_semaphore, enabled=0
[Vulkan init] extensions: name=VK_KHR_external_semaphore_fd, enabled=0
[Vulkan init] extensions: name=VK_KHR_get_memory_requirements2, enabled=1
[Vulkan init] extensions: name=VK_KHR_image_format_list, enabled=1
[Vulkan init] extensions: name=VK_KHR_imageless_framebuffer, enabled=0
[Vulkan init] extensions: name=VK_KHR_maintenance1, enabled=1
[Vulkan init] extensions: name=VK_KHR_maintenance2, enabled=0
[Vulkan init] extensions: name=VK_KHR_maintenance3, enabled=0
[Vulkan init] extensions: name=VK_KHR_multiview, enabled=1
[Vulkan init] extensions: name=VK_KHR_pipeline_executable_properties, enabled=0
[Vulkan init] extensions: name=VK_KHR_push_descriptor, enabled=0
[Vulkan init] extensions: name=VK_KHR_relaxed_block_layout, enabled=0
[Vulkan init] extensions: name=VK_KHR_sampler_mirror_clamp_to_edge, enabled=1
[Vulkan init] extensions: name=VK_KHR_sampler_ycbcr_conversion, enabled=0
[Vulkan init] extensions: name=VK_KHR_shader_atomic_int64, enabled=0
[Vulkan init] extensions: name=VK_KHR_shader_draw_parameters, enabled=0
[Vulkan init] extensions: name=VK_KHR_shader_float16_int8, enabled=0
[Vulkan init] extensions: name=VK_KHR_shader_float_controls, enabled=0
[Vulkan init] extensions: name=VK_KHR_storage_buffer_storage_class, enabled=0
[Vulkan init] extensions: name=VK_KHR_swapchain, enabled=1
[Vulkan init] extensions: name=VK_KHR_swapchain_mutable_format, enabled=0
[Vulkan init] extensions: name=VK_KHR_timeline_semaphore, enabled=0
[Vulkan init] extensions: name=VK_KHR_uniform_buffer_standard_layout, enabled=0
[Vulkan init] extensions: name=VK_KHR_variable_pointers, enabled=0
[Vulkan init] extensions: name=VK_KHR_vulkan_memory_model, enabled=0
[Vulkan init] extensions: name=VK_EXT_blend_operation_advanced, enabled=0
[Vulkan init] extensions: name=VK_EXT_buffer_device_address, enabled=0
[Vulkan init] extensions: name=VK_EXT_calibrated_timestamps, enabled=0
[Vulkan init] extensions: name=VK_EXT_conditional_rendering, enabled=0
[Vulkan init] extensions: name=VK_EXT_conservative_rasterization, enabled=0
[Vulkan init] extensions: name=VK_EXT_depth_clip_enable, enabled=0
[Vulkan init] extensions: name=VK_EXT_depth_range_unrestricted, enabled=0
[Vulkan init] extensions: name=VK_EXT_descriptor_indexing, enabled=0
[Vulkan init] extensions: name=VK_EXT_discard_rectangles, enabled=0
[Vulkan init] extensions: name=VK_EXT_display_control, enabled=0
[Vulkan init] extensions: name=VK_EXT_fragment_shader_interlock, enabled=0
[Vulkan init] extensions: name=VK_EXT_global_priority, enabled=0
[Vulkan init] extensions: name=VK_EXT_host_query_reset, enabled=0
[Vulkan init] extensions: name=VK_EXT_index_type_uint8, enabled=0
[Vulkan init] extensions: name=VK_EXT_inline_uniform_block, enabled=0
[Vulkan init] extensions: name=VK_EXT_line_rasterization, enabled=0
[Vulkan init] extensions: name=VK_EXT_memory_budget, enabled=0
[Vulkan init] extensions: name=VK_EXT_pci_bus_info, enabled=0
[Vulkan init] extensions: name=VK_EXT_pipeline_creation_feedback, enabled=0
[Vulkan init] extensions: name=VK_EXT_post_depth_coverage, enabled=0
[Vulkan init] extensions: name=VK_EXT_sample_locations, enabled=0
[Vulkan init] extensions: name=VK_EXT_sampler_filter_minmax, enabled=0
[Vulkan init] extensions: name=VK_EXT_scalar_block_layout, enabled=0
[Vulkan init] extensions: name=VK_EXT_separate_stencil_usage, enabled=0
[Vulkan init] extensions: name=VK_EXT_shader_demote_to_helper_invocation, enabled=0
[Vulkan init] extensions: name=VK_EXT_shader_subgroup_ballot, enabled=0
[Vulkan init] extensions: name=VK_EXT_shader_subgroup_vote, enabled=0
[Vulkan init] extensions: name=VK_EXT_shader_viewport_index_layer, enabled=0
[Vulkan init] extensions: name=VK_EXT_subgroup_size_control, enabled=0
[Vulkan init] extensions: name=VK_EXT_texel_buffer_alignment, enabled=0
[Vulkan init] extensions: name=VK_EXT_transform_feedback, enabled=0
[Vulkan init] extensions: name=VK_EXT_vertex_attribute_divisor, enabled=0
[Vulkan init] extensions: name=VK_EXT_ycbcr_image_arrays, enabled=0
[Vulkan init] extensions: name=VK_NV_clip_space_w_scaling, enabled=0
[Vulkan init] extensions: name=VK_NV_dedicated_allocation, enabled=0
[Vulkan init] extensions: name=VK_NV_dedicated_allocation_image_aliasing, enabled=0
[Vulkan init] extensions: name=VK_NV_device_diagnostic_checkpoints, enabled=0
[Vulkan init] extensions: name=VK_NV_fill_rectangle, enabled=0
[Vulkan init] extensions: name=VK_NV_fragment_coverage_to_color, enabled=0
[Vulkan init] extensions: name=VK_NV_framebuffer_mixed_samples, enabled=0
[Vulkan init] extensions: name=VK_NV_coverage_reduction_mode, enabled=0
[Vulkan init] extensions: name=VK_NV_geometry_shader_passthrough, enabled=0
[Vulkan init] extensions: name=VK_NV_sample_mask_override_coverage, enabled=0
[Vulkan init] extensions: name=VK_NV_shader_sm_builtins, enabled=0
[Vulkan init] extensions: name=VK_NV_shader_subgroup_partitioned, enabled=0
[Vulkan init] extensions: name=VK_NV_viewport_array2, enabled=0
[Vulkan init] extensions: name=VK_NV_viewport_swizzle, enabled=0
[Vulkan init] extensions: name=VK_NVX_device_generated_commands, enabled=0
[Vulkan init] extensions: name=VK_NVX_multiview_per_view_attributes, enabled=0
[Vulkan init] extensions: name=VK_NV_ray_tracing, enabled=0
Vulkan API version 1.1.0 (1.1.119 supported by driver)
Vulkan vendor=[NVIDIA] id=[10de]
Vulkan renderer=[GeForce GTX 1060 6GB] id=[1c03]
Vulkan device type 2
Vulkan driverversion=[0x6e148000] uint=[1846837248]
Vulkan PSO: cache data not found [path=/tmp/DefaultCompany/training-dots-hdrp/vulkan_pso_cache.bin]
Begin MonoManager ReloadAssembly
- Completed reload, in  0.067 seconds
WARNING: Shader Unsupported: 'HDRP/Lit' - All passes removed
WARNING: Shader Did you use #pragma only_renderers and omit this platform?
requesting resize 80 x 80
resizing window to 80 x 80
Desktop is 2560 x 1080 @ 60 Hz
InitializeOrResetSwapChain 80x80 hdr=0 samples=1
UnloadTime: 0.455892 ms
Fallback handler could not load library /media/data/tmp/unity/hdrp_cam_Data/Mono/libcoreclr.so
Fallback handler could not load library /media/data/tmp/unity/hdrp_cam_Data/Mono/libcoreclr.so
Fallback handler could not load library /media/data/tmp/unity/hdrp_cam_Data/Mono/libcoreclr.so
Fallback handler could not load library /media/data/tmp/unity/hdrp_cam_Data/Mono/libSystem.dylib
Fallback handler could not load library /media/data/tmp/unity/hdrp_cam_Data/Mono/libSystem.dylib.so
Fallback handler could not load library /media/data/tmp/unity/hdrp_cam_Data/Mono/libSystem.dylib
Fallback handler could not load library /media/data/tmp/unity/hdrp_cam_Data/Mono/libcoreclr.so
Fallback handler could not load library /media/data/tmp/unity/hdrp_cam_Data/Mono/libcoreclr.so
Fallback handler could not load library /media/data/tmp/unity/hdrp_cam_Data/Mono/libcoreclr.so
Fallback handler could not load library /media/data/tmp/unity/hdrp_cam_Data/Mono/libSystem.dylib
Fallback handler could not load library /media/data/tmp/unity/hdrp_cam_Data/Mono/libSystem.dylib.so
Fallback handler could not load library /media/data/tmp/unity/hdrp_cam_Data/Mono/libSystem.dylib
requesting resize 80 x 80
resizing window to 80 x 80
Desktop is 2560 x 1080 @ 60 Hz
InitializeOrResetSwapChain 80x80 hdr=0 samples=1
Setting up 2 worker threads for Enlighten.
  Thread -> id: 7f18935fe700 -> priority: 1 
  Thread -> id: 7f1892dfd700 -> priority: 1 
NullReferenceException: Object reference not set to an instance of an object
  at UnityEngine.Rendering.HighDefinition.HDProbeSystemInternal.QueryCullResults (UnityEngine.Rendering.HighDefinition.HDProbeCullState state, UnityEngine.Rendering.HighDefinition.HDProbeCullingResults& results) [0x00050] in <39762aba73064047936cd8a048e58147>:0 
  at UnityEngine.Rendering.HighDefinition.HDProbeSystem.QueryCullResults (UnityEngine.Rendering.HighDefinition.HDProbeCullState state, UnityEngine.Rendering.HighDefinition.HDProbeCullingResults& results) [0x00000] in <39762aba73064047936cd8a048e58147>:0 
  at UnityEngine.Rendering.HighDefinition.HDRenderPipeline.TryCull (UnityEngine.Camera camera, UnityEngine.Rendering.HighDefinition.HDCamera hdCamera, UnityEngine.Rendering.ScriptableRenderContext renderContext, UnityEngine.Rendering.HighDefinition.SkyManager skyManager, UnityEngine.Rendering.ScriptableCullingParameters cullingParams, UnityEngine.Rendering.HighDefinition.HDRenderPipelineAsset hdrp, UnityEngine.Rendering.HighDefinition.HDRenderPipeline+HDCullingResults& cullingResults) [0x0011a] in <39762aba73064047936cd8a048e58147>:0 
  at UnityEngine.Rendering.HighDefinition.HDRenderPipeline.Render (UnityEngine.Rendering.ScriptableRenderContext renderContext, UnityEngine.Camera[] cameras) [0x002a8] in <39762aba73064047936cd8a048e58147>:0 
  at UnityEngine.Rendering.RenderPipeline.InternalRender (UnityEngine.Rendering.ScriptableRenderContext context, UnityEngine.Camera[] cameras) [0x0001c] in <2e20a3f09314483f9ecf28d1adc29392>:0 
  at UnityEngine.Rendering.RenderPipelineManager.DoRenderLoop_Internal (UnityEngine.Rendering.RenderPipelineAsset pipe, System.IntPtr loopPtr) [0x0003d] in <2e20a3f09314483f9ecf28d1adc29392>:0 
UnityEngine.Camera:Render()
MLAgents.Sensors.CameraSensor:ObservationToTexture(Camera, Int32, Int32)
MLAgents.Sensors.CameraSensor:GetCompressedObservation()
MLAgents.GrpcExtensions:GetObservationProto(ISensor, WriteAdapter)
MLAgents.RpcCommunicator:PutObservations(String, AgentInfo, List`1)
MLAgents.Policies.RemotePolicy:RequestDecision(AgentInfo, List`1)
MLAgents.Agent:NotifyAgentDone(DoneReason)
MLAgents.Agent:OnDisable()
 
(Filename: <39762aba73064047936cd8a048e58147> Line: 0)

Vulkan PSO: Inconsistent pipeline cache header size[0], newDataSize[678036]
##utp:{"type":"MemoryLeaks","version":2,"phase":"Immediate","time":1587677058598,"processId":17033,"allocatedMemory":75156,"memoryLabels":[{"Default":40},{"Permanent":40},{"NewDelete":15487},{"Thread":800},{"Manager":1744},{"GfxDevice":11536},{"Physics":32},{"Serialization":40},{"String":553},{"DynamicArray":30792},{"Sprites":88},{"GI":12644},{"VR":2016},{"Subsystems":-656}]}

Thanks in advance Vincent!

@automata
Copy link
Contributor Author

So it seems related with occlusion culling?

@automata
Copy link
Contributor Author

OK I disabled ambient occlusion just in case and all post processing.

Run it one more time and got on logs:

Mono path[0] = '/media/data/tmp/unity/hdrp_cam5_Data/Managed'
Mono config path = '/media/data/tmp/unity/hdrp_cam5_Data/MonoBleedingEdge/etc'
PlayerConnection initialized from /media/data/tmp/unity/hdrp_cam5_Data (debug = 0)
PlayerConnection initialized network socket : 0.0.0.0 55044
Multi-casting "[IP] 192.168.0.10 [Port] 55044 [Flags] 2 [Guid] 3739687994 [EditorId] 4223096421 [Version] 1048832 [Id] LinuxPlayer(192.168.0.10) [Debug] 0 [PackageName] LinuxPlayer [ProjectName] training-dots-hdrp" to [225.0.0.222:54997]...
Started listening to [0.0.0.0:55044]
Preloaded 'lib_burst_generated.so'
Preloaded 'libgrpc_csharp_ext.x64.so'
PlayerConnection already initialized - listening to [0.0.0.0:55044]
Display 0 'LG ULTRAWIDE 29"': 2560x1080 (primary device).
Desktop is 2560 x 1080 @ 60 Hz
Invalid initial resolution 80 x 80 - forcing to 100 x 100
[Vulkan init] extensions: count=18
[Vulkan init] extensions: name=VK_KHR_device_group_creation, enabled=0
[Vulkan init] extensions: name=VK_KHR_external_fence_capabilities, enabled=0
[Vulkan init] extensions: name=VK_KHR_external_memory_capabilities, enabled=0
[Vulkan init] extensions: name=VK_KHR_external_semaphore_capabilities, enabled=0
[Vulkan init] extensions: name=VK_KHR_get_display_properties2, enabled=0
[Vulkan init] extensions: name=VK_KHR_get_physical_device_properties2, enabled=0
[Vulkan init] extensions: name=VK_KHR_get_surface_capabilities2, enabled=0
[Vulkan init] extensions: name=VK_KHR_surface, enabled=1
[Vulkan init] extensions: name=VK_KHR_surface_protected_capabilities, enabled=0
[Vulkan init] extensions: name=VK_KHR_wayland_surface, enabled=1
[Vulkan init] extensions: name=VK_KHR_xcb_surface, enabled=0
[Vulkan init] extensions: name=VK_KHR_xlib_surface, enabled=1
[Vulkan init] extensions: name=VK_KHR_display, enabled=1
[Vulkan init] extensions: name=VK_EXT_direct_mode_display, enabled=0
[Vulkan init] extensions: name=VK_EXT_acquire_xlib_display, enabled=0
[Vulkan init] extensions: name=VK_EXT_display_surface_counter, enabled=0
[Vulkan init] extensions: name=VK_EXT_debug_report, enabled=1
[Vulkan init] extensions: name=VK_EXT_debug_utils, enabled=0
Vulkan detection: 2
Initialize engine version: 2019.3.10f1 (5968d7f82152)
[Subsystems] Discovering subsystems at path /media/data/tmp/unity/hdrp_cam5_Data/UnitySubsystems
GfxDevice: creating device client; threaded=1
[Vulkan init] extensions: count=18
[Vulkan init] extensions: name=VK_KHR_device_group_creation, enabled=0
[Vulkan init] extensions: name=VK_KHR_external_fence_capabilities, enabled=0
[Vulkan init] extensions: name=VK_KHR_external_memory_capabilities, enabled=0
[Vulkan init] extensions: name=VK_KHR_external_semaphore_capabilities, enabled=0
[Vulkan init] extensions: name=VK_KHR_get_display_properties2, enabled=0
[Vulkan init] extensions: name=VK_KHR_get_physical_device_properties2, enabled=0
[Vulkan init] extensions: name=VK_KHR_get_surface_capabilities2, enabled=0
[Vulkan init] extensions: name=VK_KHR_surface, enabled=1
[Vulkan init] extensions: name=VK_KHR_surface_protected_capabilities, enabled=0
[Vulkan init] extensions: name=VK_KHR_wayland_surface, enabled=1
[Vulkan init] extensions: name=VK_KHR_xcb_surface, enabled=0
[Vulkan init] extensions: name=VK_KHR_xlib_surface, enabled=1
[Vulkan init] extensions: name=VK_KHR_display, enabled=1
[Vulkan init] extensions: name=VK_EXT_direct_mode_display, enabled=0
[Vulkan init] extensions: name=VK_EXT_acquire_xlib_display, enabled=0
[Vulkan init] extensions: name=VK_EXT_display_surface_counter, enabled=0
[Vulkan init] extensions: name=VK_EXT_debug_report, enabled=1
[Vulkan init] extensions: name=VK_EXT_debug_utils, enabled=0
[Vulkan init] extensions: count=89
[Vulkan init] extensions: name=VK_KHR_8bit_storage, enabled=0
[Vulkan init] extensions: name=VK_KHR_16bit_storage, enabled=0
[Vulkan init] extensions: name=VK_KHR_bind_memory2, enabled=0
[Vulkan init] extensions: name=VK_KHR_create_renderpass2, enabled=0
[Vulkan init] extensions: name=VK_KHR_dedicated_allocation, enabled=1
[Vulkan init] extensions: name=VK_KHR_depth_stencil_resolve, enabled=0
[Vulkan init] extensions: name=VK_KHR_descriptor_update_template, enabled=1
[Vulkan init] extensions: name=VK_KHR_device_group, enabled=0
[Vulkan init] extensions: name=VK_KHR_draw_indirect_count, enabled=0
[Vulkan init] extensions: name=VK_KHR_driver_properties, enabled=0
[Vulkan init] extensions: name=VK_KHR_external_fence, enabled=0
[Vulkan init] extensions: name=VK_KHR_external_fence_fd, enabled=0
[Vulkan init] extensions: name=VK_KHR_external_memory, enabled=0
[Vulkan init] extensions: name=VK_KHR_external_memory_fd, enabled=0
[Vulkan init] extensions: name=VK_KHR_external_semaphore, enabled=0
[Vulkan init] extensions: name=VK_KHR_external_semaphore_fd, enabled=0
[Vulkan init] extensions: name=VK_KHR_get_memory_requirements2, enabled=1
[Vulkan init] extensions: name=VK_KHR_image_format_list, enabled=1
[Vulkan init] extensions: name=VK_KHR_imageless_framebuffer, enabled=0
[Vulkan init] extensions: name=VK_KHR_maintenance1, enabled=1
[Vulkan init] extensions: name=VK_KHR_maintenance2, enabled=0
[Vulkan init] extensions: name=VK_KHR_maintenance3, enabled=0
[Vulkan init] extensions: name=VK_KHR_multiview, enabled=1
[Vulkan init] extensions: name=VK_KHR_pipeline_executable_properties, enabled=0
[Vulkan init] extensions: name=VK_KHR_push_descriptor, enabled=0
[Vulkan init] extensions: name=VK_KHR_relaxed_block_layout, enabled=0
[Vulkan init] extensions: name=VK_KHR_sampler_mirror_clamp_to_edge, enabled=1
[Vulkan init] extensions: name=VK_KHR_sampler_ycbcr_conversion, enabled=0
[Vulkan init] extensions: name=VK_KHR_shader_atomic_int64, enabled=0
[Vulkan init] extensions: name=VK_KHR_shader_draw_parameters, enabled=0
[Vulkan init] extensions: name=VK_KHR_shader_float16_int8, enabled=0
[Vulkan init] extensions: name=VK_KHR_shader_float_controls, enabled=0
[Vulkan init] extensions: name=VK_KHR_storage_buffer_storage_class, enabled=0
[Vulkan init] extensions: name=VK_KHR_swapchain, enabled=1
[Vulkan init] extensions: name=VK_KHR_swapchain_mutable_format, enabled=0
[Vulkan init] extensions: name=VK_KHR_timeline_semaphore, enabled=0
[Vulkan init] extensions: name=VK_KHR_uniform_buffer_standard_layout, enabled=0
[Vulkan init] extensions: name=VK_KHR_variable_pointers, enabled=0
[Vulkan init] extensions: name=VK_KHR_vulkan_memory_model, enabled=0
[Vulkan init] extensions: name=VK_EXT_blend_operation_advanced, enabled=0
[Vulkan init] extensions: name=VK_EXT_buffer_device_address, enabled=0
[Vulkan init] extensions: name=VK_EXT_calibrated_timestamps, enabled=0
[Vulkan init] extensions: name=VK_EXT_conditional_rendering, enabled=0
[Vulkan init] extensions: name=VK_EXT_conservative_rasterization, enabled=0
[Vulkan init] extensions: name=VK_EXT_depth_clip_enable, enabled=0
[Vulkan init] extensions: name=VK_EXT_depth_range_unrestricted, enabled=0
[Vulkan init] extensions: name=VK_EXT_descriptor_indexing, enabled=0
[Vulkan init] extensions: name=VK_EXT_discard_rectangles, enabled=0
[Vulkan init] extensions: name=VK_EXT_display_control, enabled=0
[Vulkan init] extensions: name=VK_EXT_fragment_shader_interlock, enabled=0
[Vulkan init] extensions: name=VK_EXT_global_priority, enabled=0
[Vulkan init] extensions: name=VK_EXT_host_query_reset, enabled=0
[Vulkan init] extensions: name=VK_EXT_index_type_uint8, enabled=0
[Vulkan init] extensions: name=VK_EXT_inline_uniform_block, enabled=0
[Vulkan init] extensions: name=VK_EXT_line_rasterization, enabled=0
[Vulkan init] extensions: name=VK_EXT_memory_budget, enabled=0
[Vulkan init] extensions: name=VK_EXT_pci_bus_info, enabled=0
[Vulkan init] extensions: name=VK_EXT_pipeline_creation_feedback, enabled=0
[Vulkan init] extensions: name=VK_EXT_post_depth_coverage, enabled=0
[Vulkan init] extensions: name=VK_EXT_sample_locations, enabled=0
[Vulkan init] extensions: name=VK_EXT_sampler_filter_minmax, enabled=0
[Vulkan init] extensions: name=VK_EXT_scalar_block_layout, enabled=0
[Vulkan init] extensions: name=VK_EXT_separate_stencil_usage, enabled=0
[Vulkan init] extensions: name=VK_EXT_shader_demote_to_helper_invocation, enabled=0
[Vulkan init] extensions: name=VK_EXT_shader_subgroup_ballot, enabled=0
[Vulkan init] extensions: name=VK_EXT_shader_subgroup_vote, enabled=0
[Vulkan init] extensions: name=VK_EXT_shader_viewport_index_layer, enabled=0
[Vulkan init] extensions: name=VK_EXT_subgroup_size_control, enabled=0
[Vulkan init] extensions: name=VK_EXT_texel_buffer_alignment, enabled=0
[Vulkan init] extensions: name=VK_EXT_transform_feedback, enabled=0
[Vulkan init] extensions: name=VK_EXT_vertex_attribute_divisor, enabled=0
[Vulkan init] extensions: name=VK_EXT_ycbcr_image_arrays, enabled=0
[Vulkan init] extensions: name=VK_NV_clip_space_w_scaling, enabled=0
[Vulkan init] extensions: name=VK_NV_dedicated_allocation, enabled=0
[Vulkan init] extensions: name=VK_NV_dedicated_allocation_image_aliasing, enabled=0
[Vulkan init] extensions: name=VK_NV_device_diagnostic_checkpoints, enabled=0
[Vulkan init] extensions: name=VK_NV_fill_rectangle, enabled=0
[Vulkan init] extensions: name=VK_NV_fragment_coverage_to_color, enabled=0
[Vulkan init] extensions: name=VK_NV_framebuffer_mixed_samples, enabled=0
[Vulkan init] extensions: name=VK_NV_coverage_reduction_mode, enabled=0
[Vulkan init] extensions: name=VK_NV_geometry_shader_passthrough, enabled=0
[Vulkan init] extensions: name=VK_NV_sample_mask_override_coverage, enabled=0
[Vulkan init] extensions: name=VK_NV_shader_sm_builtins, enabled=0
[Vulkan init] extensions: name=VK_NV_shader_subgroup_partitioned, enabled=0
[Vulkan init] extensions: name=VK_NV_viewport_array2, enabled=0
[Vulkan init] extensions: name=VK_NV_viewport_swizzle, enabled=0
[Vulkan init] extensions: name=VK_NVX_device_generated_commands, enabled=0
[Vulkan init] extensions: name=VK_NVX_multiview_per_view_attributes, enabled=0
[Vulkan init] extensions: name=VK_NV_ray_tracing, enabled=0
Vulkan API version 1.1.0 (1.1.119 supported by driver)
Vulkan vendor=[NVIDIA] id=[10de]
Vulkan renderer=[GeForce GTX 1060 6GB] id=[1c03]
Vulkan device type 2
Vulkan driverversion=[0x6e148000] uint=[1846837248]
Vulkan PSO: cache data not found [path=/tmp/DefaultCompany/training-dots-hdrp/vulkan_pso_cache.bin]
Begin MonoManager ReloadAssembly
- Completed reload, in  0.719 seconds
WARNING: Shader Unsupported: 'HDRP/Lit' - All passes removed
WARNING: Shader Did you use #pragma only_renderers and omit this platform?
requesting resize 80 x 80
resizing window to 80 x 80
Desktop is 2560 x 1080 @ 60 Hz
InitializeOrResetSwapChain 80x80 hdr=0 samples=1
UnloadTime: 0.484660 ms
Fallback handler could not load library /media/data/tmp/unity/hdrp_cam5_Data/Mono/libcoreclr.so
Fallback handler could not load library /media/data/tmp/unity/hdrp_cam5_Data/Mono/libcoreclr.so
Fallback handler could not load library /media/data/tmp/unity/hdrp_cam5_Data/Mono/libcoreclr.so
Fallback handler could not load library /media/data/tmp/unity/hdrp_cam5_Data/Mono/libSystem.dylib
Fallback handler could not load library /media/data/tmp/unity/hdrp_cam5_Data/Mono/libSystem.dylib.so
Fallback handler could not load library /media/data/tmp/unity/hdrp_cam5_Data/Mono/libSystem.dylib
Fallback handler could not load library /media/data/tmp/unity/hdrp_cam5_Data/Mono/libcoreclr.so
Fallback handler could not load library /media/data/tmp/unity/hdrp_cam5_Data/Mono/libcoreclr.so
Fallback handler could not load library /media/data/tmp/unity/hdrp_cam5_Data/Mono/libcoreclr.so
Fallback handler could not load library /media/data/tmp/unity/hdrp_cam5_Data/Mono/libSystem.dylib
Fallback handler could not load library /media/data/tmp/unity/hdrp_cam5_Data/Mono/libSystem.dylib.so
Fallback handler could not load library /media/data/tmp/unity/hdrp_cam5_Data/Mono/libSystem.dylib
requesting resize 80 x 80
resizing window to 80 x 80
Desktop is 2560 x 1080 @ 60 Hz
InitializeOrResetSwapChain 80x80 hdr=0 samples=1
Compute dispatch: missing UAV ID %d 
(Filename:  Line: 0)

Compute dispatch: missing texture ID %d 
(Filename:  Line: 0)

Compute dispatch: missing texture ID %d 
(Filename:  Line: 0)

Compute dispatch: missing texture ID %d 
(Filename:  Line: 0)

Compute dispatch: missing texture ID %d 
(Filename:  Line: 0)

Compute dispatch: missing texture ID %d 
(Filename:  Line: 0)

Compute dispatch: missing texture ID %d 
(Filename:  Line: 0)

Compute dispatch: missing UAV ID %d 
(Filename:  Line: 0)

Compute dispatch: missing texture ID %d 
(Filename:  Line: 0)

Compute dispatch: missing texture ID %d 
(Filename:  Line: 0)

Compute dispatch: missing texture ID %d 
(Filename:  Line: 0)

Compute dispatch: missing texture ID %d 
(Filename:  Line: 0)

Compute dispatch: missing texture ID %d 
(Filename:  Line: 0)

Compute dispatch: missing UAV ID %d 
(Filename:  Line: 0)

Compute dispatch: missing texture ID %d 
(Filename:  Line: 0)

Compute dispatch: missing texture ID %d 
(Filename:  Line: 0)

Compute dispatch: missing texture ID %d 
(Filename:  Line: 0)

Compute dispatch: missing texture ID %d 
(Filename:  Line: 0)

Compute dispatch: missing UAV ID %d 
(Filename:  Line: 0)

Compute dispatch: missing texture ID %d 
(Filename:  Line: 0)

Compute dispatch: missing texture ID %d 
(Filename:  Line: 0)

Compute dispatch: missing texture ID %d 
(Filename:  Line: 0)

Compute dispatch: missing UAV ID %d 
(Filename:  Line: 0)

Compute dispatch: missing texture ID %d 
(Filename:  Line: 0)

Compute dispatch: missing texture ID %d 
(Filename:  Line: 0)

Compute dispatch: missing texture ID %d 
(Filename:  Line: 0)

Compute dispatch: missing texture ID %d 
(Filename:  Line: 0)

Setting up 2 worker threads for Enlighten.
  Thread -> id: 7fc629979700 -> priority: 1 
  Thread -> id: 7fc629178700 -> priority: 1 
AssertionException: Assertion failure. Value was Null
Expected: Value was not Null
Culling was not prepared, please prepare cull before performing it.
  at UnityEngine.Assertions.Assert.Fail (System.String message, System.String userMessage) [0x0003c] in /home/builduser/buildslave/unity/build/Runtime/Export/Assertions/Assert/AssertBase.cs:29 
  at UnityEngine.Assertions.Assert.IsNotNull[T] (T value, System.String message) [0x00042] in /home/builduser/buildslave/unity/build/Runtime/Export/Assertions/Assert/AssertNull.cs:50 
  at UnityEngine.Rendering.HighDefinition.HDProbeSystemInternal.QueryCullResults (UnityEngine.Rendering.HighDefinition.HDProbeCullState state, UnityEngine.Rendering.HighDefinition.HDProbeCullingResults& results) [0x00001] in /media/data/vilson/src/extend/continuum/render/kb/expt/training-dots-hdrp/Library/PackageCache/com.unity.render-pipelines.high-definition@7.3.1/Runtime/Lighting/Reflection/HDProbeSystem.cs:263 
  at UnityEngine.Rendering.HighDefinition.HDProbeSystem.QueryCullResults (UnityEngine.Rendering.HighDefinition.HDProbeCullState state, UnityEngine.Rendering.HighDefinition.HDProbeCullingResults& results) [0x00000] in /media/data/vilson/src/extend/continuum/render/kb/expt/training-dots-hdrp/Library/PackageCache/com.unity.render-pipelines.high-definition@7.3.1/Runtime/Lighting/Reflection/HDProbeSystem.cs:80 
  at UnityEngine.Rendering.HighDefinition.HDRenderPipeline.TryCull (UnityEngine.Camera camera, UnityEngine.Rendering.HighDefinition.HDCamera hdCamera, UnityEngine.Rendering.ScriptableRenderContext renderContext, UnityEngine.Rendering.HighDefinition.SkyManager skyManager, UnityEngine.Rendering.ScriptableCullingParameters cullingParams, UnityEngine.Rendering.HighDefinition.HDRenderPipelineAsset hdrp, UnityEngine.Rendering.HighDefinition.HDRenderPipeline+HDCullingResults& cullingResults) [0x0013c] in /media/data/vilson/src/extend/continuum/render/kb/expt/training-dots-hdrp/Library/PackageCache/com.unity.render-pipelines.high-definition@7.3.1/Runtime/RenderPipeline/HDRenderPipeline.cs:2683 
  at UnityEngine.Rendering.HighDefinition.HDRenderPipeline.Render (UnityEngine.Rendering.ScriptableRenderContext renderContext, UnityEngine.Camera[] cameras) [0x00315] in /media/data/vilson/src/extend/continuum/render/kb/expt/training-dots-hdrp/Library/PackageCache/com.unity.render-pipelines.high-definition@7.3.1/Runtime/RenderPipeline/HDRenderPipeline.cs:1311 
  at UnityEngine.Rendering.RenderPipeline.InternalRender (UnityEngine.Rendering.ScriptableRenderContext context, UnityEngine.Camera[] cameras) [0x0001c] in /home/builduser/buildslave/unity/build/Runtime/Export/RenderPipeline/RenderPipeline.cs:34 
  at UnityEngine.Rendering.RenderPipelineManager.DoRenderLoop_Internal (UnityEngine.Rendering.RenderPipelineAsset pipe, System.IntPtr loopPtr) [0x00038] in /home/builduser/buildslave/unity/build/Runtime/Export/RenderPipeline/RenderPipelineManager.cs:86 
UnityEngine.Camera:Render()
MLAgents.Sensors.CameraSensor:ObservationToTexture(Camera, Int32, Int32) (at /media/data/vilson/src/extend/continuum/render/kb/expt/training-dots-hdrp/com.unity.ml-agents/Runtime/Sensors/CameraSensor.cs:145)
MLAgents.Sensors.CameraSensor:GetCompressedObservation() (at /media/data/vilson/src/extend/continuum/render/kb/expt/training-dots-hdrp/com.unity.ml-agents/Runtime/Sensors/CameraSensor.cs:85)
MLAgents.GrpcExtensions:GetObservationProto(ISensor, WriteAdapter) (at /media/data/vilson/src/extend/continuum/render/kb/expt/training-dots-hdrp/com.unity.ml-agents/Runtime/Communicator/GrpcExtensions.cs:232)
MLAgents.RpcCommunicator:PutObservations(String, AgentInfo, List`1) (at /media/data/vilson/src/extend/continuum/render/kb/expt/training-dots-hdrp/com.unity.ml-agents/Runtime/Communicator/RpcCommunicator.cs:287)
MLAgents.Policies.RemotePolicy:RequestDecision(AgentInfo, List`1) (at /media/data/vilson/src/extend/continuum/render/kb/expt/training-dots-hdrp/com.unity.ml-agents/Runtime/Policies/RemotePolicy.cs:33)
MLAgents.Agent:NotifyAgentDone(DoneReason) (at /media/data/vilson/src/extend/continuum/render/kb/expt/training-dots-hdrp/com.unity.ml-agents/Runtime/Agent.cs:324)
MLAgents.Agent:OnDisable() (at /media/data/vilson/src/extend/continuum/render/kb/expt/training-dots-hdrp/com.unity.ml-agents/Runtime/Agent.cs:311)
 
(Filename: /home/builduser/buildslave/unity/build/Runtime/Export/Assertions/Assert/AssertBase.cs Line: 29)

Vulkan PSO: Inconsistent pipeline cache header size[0], newDataSize[673224]
##utp:{"type":"MemoryLeaks","version":2,"phase":"Immediate","time":1587678800201,"processId":25753,"allocatedMemory":75157,"memoryLabels":[{"Default":40},{"Permanent":40},{"NewDelete":15487},{"Thread":800},{"Manager":1744},{"GfxDevice":11536},{"Physics":32},{"Serialization":40},{"String":554},{"DynamicArray":30792},{"Sprites":88},{"GI":12644},{"VR":2016},{"Subsystems":-656}]}

While searching about that new error Compute dispatch: missing texture ID %d (Filename: Line: 0) I got this https://forum.unity.com/threads/screen-setresolution-causing-materials-to-stop-rendering.804681/. Maybe related?

@vincentpierre
Copy link
Contributor

Maybe this is due to our use Screen.SetResolution in the code. Can you try commenting this line out and try again?

@automata
Copy link
Contributor Author

Great catch Vincent! I commented that out and now it works perfectly:

Screenshot from 2020-04-24 08-49-32

The error message related with culling is still there but no more complains about missing textures.

Inspired by your suggestion I tried to change settings to allow resizable window like:

Screenshot from 2020-04-24 08-45-52

However that didn't work neither.

I put all the post processing back and your solution works.

Thank you!

@automata
Copy link
Contributor Author

Should I open a PR for that?

@surfnerd
Copy link
Contributor

Hi @automata,
This looks like a graphics bug within unity itself, not ML-Agents. We are working with our graphics team to see if it's already fixed in a newer version, or if we can file a bug for this so they can fix it. Thank you for finding this!

@vincentpierre
Copy link
Contributor

@surfnerd , We could do

# if not PLATFORM_LINUX and not EDITOR and VERSION_2019_3
Screen.SetResolution(width, height)
#endif

to make sure we do not set the resolution on this platform.
We could also not set the screen resolution by default in learn.py

vincentpierre added a commit that referenced this issue Apr 27, 2020
@vincentpierre
Removed the default for width and height of the executable training. …
6e6e1cb 
…This is to help relove #3835 since setting the screen resolution on Linux 2019.3 can cause issues.
@vincentpierre vincentpierre mentioned this issue Apr 27, 2020
Merged
10 tasks
vincentpierre added a commit that referenced this issue Apr 27, 2020
@vincentpierre
Removed the default for width and height of the executable training. (#…
c69ad6b 
…3867)

* Removed the default for width and height of the executable training. This is to help relove #3835 since setting the screen resolution on Linux 2019.3 can cause issues.

* Editing the changelog

* Making fields in EngineConfig optional
@vincentpierre
Copy link
Contributor

Tracking this bug internally : MLA-961
In the mean time, we will now not set the resolution by default when doing training #3867

@github-actions
Copy link

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Jan 30, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@vincentpierre vincentpierre

Labels
bug Issue describes a potential bug in ml-agents.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@automata @surfnerd @vincentpierre