Trouble identifying cause of segfault

help
1

Hiya, I have an obscure segfault error when running cargo test, but am having trouble figuring out why it's happening. The source code is closed else I'd link to it, but still hoping someone would be able to help / suggest where to look based on the crashing output.

What cargo test looks like...

when it works (my user)

Environment:

  • Ubuntu 18.04, with a proper X display
  • rustc 1.28.0-nightly (b68432d56 2018-06-12)
$ cargo test
    Finished dev [unoptimized + debuginfo] target(s) in 0.25s
     Running /home/azriel/work/gitlab/azriel91/autexousious/target/debug/deps/game_play-482ef6f20580cf49

running 2 tests
[amethyst::app][INFO] Initializing Amethyst...
[amethyst::app][INFO] Version: 0.7.0
[amethyst::app][INFO] Platform: x86_64-unknown-linux-gnu
[amethyst::app][INFO] Amethyst git commit: 00fa904612be5609121981a9663a7aea097e3a55
[amethyst::app][INFO] Rustc version: 1.28.0-nightly Nightly
[amethyst::app][INFO] Rustc git commit: b68432d560c7c6f1e738b27e49d271a2a778f898
test game_play_bundle::test::bundle_build_should_succeed ... ok
[gfx_device_gl::info][INFO] parsing version '4.2 (Core Profile) Mesa 18.0.0-rc5'
[gfx_device_gl::info][INFO] parsing version '4.20'
[gfx_device_gl][INFO] Info { platform_name: PlatformName { vendor: "Intel Open Source Technology Center", renderer: "Mesa DRI Intel(R) Ivybridge Mobile " }, version: 4.2, shading_language: 4.20, extensions: {"GL_ARB_internalformat_query2", "GL_ARB_shader_texture_lod", "GL_ARB_texture_rg", "GL_ARB_pipeline_statistics_query", "GL_EXT_framebuffer_blit", "GL_ARB_shading_language_packing", "GL_IBM_multimode_draw_arrays", "GL_ARB_explicit_uniform_location", "GL_ARB_texture_gather", "GL_ANGLE_texture_compression_dxt3", "GL_ARB_seamless_cubemap_per_texture", "GL_KHR_robustness", "GL_ARB_seamless_cube_map", "GL_ARB_explicit_attrib_location", "GL_ARB_draw_buffers_blend", "GL_ARB_vertex_attrib_binding", "GL_EXT_texture_swizzle", "GL_ARB_map_buffer_range", "GL_ARB_sync", "GL_ARB_derivative_control", "GL_ARB_shader_image_load_store", "GL_ARB_blend_func_extended", "GL_ARB_compute_shader", "GL_ARB_shader_storage_buffer_object", "GL_ARB_pixel_buffer_object", "GL_ARB_texture_buffer_range", "GL_ARB_multi_draw_indirect", "GL_ARB_conditional_render_inverted", "GL_EXT_texture_shared_exponent", "GL_MESA_texture_signed_rgba", "GL_ARB_half_float_pixel", "GL_ARB_shader_image_size", "GL_ARB_texture_query_levels", "GL_ARB_texture_compression_bptc", "GL_EXT_draw_instanced", "GL_ARB_texture_storage", "GL_KHR_context_flush_control", "GL_EXT_texture_sRGB", "GL_ARB_texture_buffer_object", "GL_EXT_packed_depth_stencil", "GL_ARB_transform_feedback3", "GL_S3_s3tc", "GL_ARB_buffer_storage", "GL_ARB_program_interface_query", "GL_ARB_draw_instanced", "GL_AMD_vertex_shader_viewport_index", "GL_ARB_draw_buffers", "GL_ARB_draw_elements_base_vertex", "GL_ARB_provoking_vertex", "GL_ARB_texture_cube_map_array", "GL_ARB_texture_query_lod", "GL_ARB_shader_precision", "GL_MESA_shader_integer_functions", "GL_KHR_debug", "GL_ARB_texture_compression_rgtc", "GL_ARB_base_instance", "GL_ARB_vertex_shader", "GL_EXT_shader_integer_mix", "GL_ARB_ES3_compatibility", "GL_ARB_invalidate_subdata", "GL_KHR_blend_equation_advanced", "GL_ARB_clip_control", "GL_ARB_robustness", "GL_ATI_blend_equation_separate", "GL_EXT_framebuffer_multisample_blit_scaled", "GL_NV_texture_barrier", "GL_ARB_copy_image", "GL_EXT_shader_samples_identical", "GL_ARB_depth_clamp", "GL_ARB_shader_bit_encoding", "GL_ARB_timer_query", "GL_ARB_transform_feedback_overflow_query", "GL_ARB_shader_atomic_counters", "GL_EXT_texture_compression_dxt1", "GL_ARB_instanced_arrays", "GL_ARB_conservative_depth", "GL_EXT_polygon_offset_clamp", "GL_ARB_vertex_type_10f_11f_11f_rev", "GL_EXT_texture_sRGB_decode", "GL_ARB_texture_filter_anisotropic", "GL_EXT_texture_integer", "GL_NV_conditional_render", "GL_ARB_framebuffer_object", "GL_ARB_point_sprite", "GL_ARB_depth_buffer_float", "GL_AMD_shader_trinary_minmax", "GL_EXT_timer_query", "GL_AMD_vertex_shader_layer", "GL_ARB_framebuffer_sRGB", "GL_ARB_uniform_buffer_object", "GL_KHR_no_error", "GL_ARB_shader_texture_image_samples", "GL_ARB_ES2_compatibility", "GL_ATI_texture_float", "GL_ARB_shading_language_420pack", "GL_NV_packed_depth_stencil", "GL_EXT_blend_equation_separate", "GL_ARB_half_float_vertex", "GL_ARB_arrays_of_arrays", "GL_ARB_viewport_array", "GL_ARB_indirect_parameters", "GL_ARB_get_program_binary", "GL_EXT_provoking_vertex", "GL_EXT_abgr", "GL_MESA_pack_invert", "GL_ARB_shader_viewport_layer_array", "GL_ARB_texture_rgb10_a2ui", "GL_EXT_pixel_buffer_object", "GL_ARB_shader_objects", "GL_EXT_texture_array", "GL_ARB_transform_feedback_instanced", "GL_EXT_texture_compression_rgtc", "GL_ARB_enhanced_layouts", "GL_ARB_polygon_offset_clamp", "GL_AMD_conservative_depth", "GL_ARB_draw_indirect", "GL_ARB_vertex_attrib_64bit", "GL_ARB_map_buffer_alignment", "GL_3DFX_texture_compression_FXT1", "GL_EXT_framebuffer_multisample", "GL_EXT_transform_feedback", "GL_ARB_tessellation_shader", "GL_ARB_framebuffer_no_attachments", "GL_ARB_clear_texture", "GL_ARB_debug_output", "GL_ARB_texture_non_power_of_two", "GL_ANGLE_texture_compression_dxt5", "GL_EXT_packed_float", "GL_ARB_texture_mirror_clamp_to_edge", "GL_ARB_compressed_texture_pixel_storage", "GL_ARB_direct_state_access", "GL_ARB_gpu_shader5", "GL_ARB_internalformat_query", "GL_EXT_texture_snorm", "GL_ARB_fragment_coord_conventions", "GL_ARB_sampler_objects", "GL_ARB_transform_feedback2", "GL_ARB_vertex_array_bgra", "GL_ARB_texture_rectangle", "GL_ARB_fragment_layer_viewport", "GL_ARB_get_texture_sub_image", "GL_AMD_draw_buffers_blend", "GL_ARB_separate_shader_objects", "GL_EXT_draw_buffers2", "GL_ARB_multi_bind", "GL_ARB_texture_float", "GL_OES_EGL_image", "GL_APPLE_object_purgeable", "GL_ARB_texture_barrier", "GL_ARB_texture_storage_multisample", "GL_ARB_occlusion_query2", "GL_ARB_shader_atomic_counter_ops", "GL_ARB_sample_shading", "GL_ARB_texture_buffer_object_rgb32", "GL_ARB_shader_subroutine", "GL_ARB_vertex_array_object", "GL_EXT_texture_filter_anisotropic", "GL_INTEL_performance_query", "GL_ARB_texture_multisample", "GL_EXT_vertex_array_bgra", "GL_ARB_shader_group_vote", "GL_EXT_texture_compression_s3tc", "GL_NV_depth_clamp", "GL_ARB_gpu_shader_fp64", "GL_ARB_clear_buffer_object", "GL_ARB_shader_clock", "GL_ARB_shader_draw_parameters", "GL_EXT_framebuffer_sRGB", "GL_ARB_texture_view", "GL_ARB_vertex_type_2_10_10_10_rev", "GL_ARB_texture_swizzle", "GL_ARB_cull_distance", "GL_AMD_seamless_cubemap_per_texture", "GL_ARB_fragment_shader", "GL_ARB_copy_buffer"} }

... lots of (probably) non-useful messages removed ...

[amethyst_assets::storage][DEBUG] "renderer::Texture": Asset "/home/azriel/work/gitlab/azriel91/autexousious/assets/test/object/character/bat/bat_grey.png" (handle id: Handle { id: 7, marker: PhantomData }) has been loaded successfully
[game_play::character_entity_spawner][DEBUG] Retrieving character with handle: `Handle { id: 0, marker: PhantomData }`
[amethyst::app][INFO] Engine is shutting down
test character_entity_spawner::test::spawn_for_player_creates_entity_with_object_components ... ok

test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

   Doc-tests game_play

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out
when it doesn't work (CI user)
  • Ubuntu 18.04 (same machine), using XVFB
$ cargo +nightly test
    Finished dev [unoptimized + debuginfo] target(s) in 0.25s
     Running /home/gitlab-runner/builds/28120612/1/azriel91/autexousious/target/debug/deps/game_play-482ef6f20580cf49

running 2 tests
[amethyst::app][INFO] Initializing Amethyst...
[amethyst::app][INFO] Version: 0.7.0
[amethyst::app][INFO] Platform: x86_64-unknown-linux-gnu
[amethyst::app][INFO] Amethyst git commit: 00fa904612be5609121981a9663a7aea097e3a55
[amethyst::app][INFO] Rustc version: 1.28.0-nightly Nightly
[amethyst::app][INFO] Rustc git commit: b68432d560c7c6f1e738b27e49d271a2a778f898
test game_play_bundle::test::bundle_build_should_succeed ... ok
[gfx_device_gl::info][INFO] parsing version '3.3 (Core Profile) Mesa 18.0.0-rc5'
[gfx_device_gl::info][INFO] parsing version '3.30'
[gfx_device_gl][INFO] Info { platform_name: PlatformName { vendor: "VMware, Inc.", renderer: "llvmpipe (LLVM 6.0, 256 bits)" }, version: 3.3, shading_language: 3.30, extensions: {"GL_ARB_clear_buffer_object", "GL_ARB_texture_query_lod", "GL_S3_s3tc", "GL_KHR_no_error", "GL_ARB_draw_elements_base_vertex", "GL_ARB_stencil_texturing", "GL_EXT_polygon_offset_clamp", "GL_ARB_conditional_render_inverted", "GL_ARB_depth_buffer_float", "GL_ARB_texture_rectangle", "GL_EXT_texture_array", "GL_ARB_copy_image", "GL_ARB_pixel_buffer_object", "GL_AMD_shader_stencil_export", "GL_ARB_framebuffer_sRGB", "GL_ARB_map_buffer_alignment", "GL_ARB_provoking_vertex", "GL_ARB_shading_language_420pack", "GL_EXT_packed_float", "GL_MESA_shader_integer_functions", "GL_KHR_debug", "GL_ARB_uniform_buffer_object", "GL_ARB_base_instance", "GL_EXT_abgr", "GL_ARB_shader_stencil_export", "GL_ARB_timer_query", "GL_EXT_texture_mirror_clamp", "GL_AMD_shader_trinary_minmax", "GL_ARB_multi_bind", "GL_ARB_transform_feedback_overflow_query", "GL_EXT_texture_sRGB", "GL_ARB_fragment_shader", "GL_ATI_texture_float", "GL_ARB_transform_feedback3", "GL_ARB_seamless_cubemap_per_texture", "GL_ARB_enhanced_layouts", "GL_EXT_texture_compression_rgtc", "GL_ARB_draw_instanced", "GL_ARB_internalformat_query", "GL_ARB_viewport_array", "GL_ARB_shader_objects", "GL_KHR_context_flush_control", "GL_ARB_gpu_shader_int64", "GL_ARB_texture_compression_rgtc", "GL_ARB_texture_rg", "GL_ARB_vertex_array_bgra", "GL_EXT_provoking_vertex", "GL_ARB_separate_shader_objects", "GL_ARB_get_program_binary", "GL_ARB_robustness", "GL_ARB_cull_distance", "GL_ARB_fragment_layer_viewport", "GL_ARB_buffer_storage", "GL_ANGLE_texture_compression_dxt5", "GL_ARB_texture_non_power_of_two", "GL_NV_depth_clamp", "GL_EXT_shader_integer_mix", "GL_AMD_vertex_shader_layer", "GL_EXT_draw_instanced", "GL_ARB_copy_buffer", "GL_ARB_texture_view", "GL_ATI_blend_equation_separate", "GL_ARB_vertex_attrib_64bit", "GL_ARB_point_sprite", "GL_ARB_get_texture_sub_image", "GL_EXT_packed_depth_stencil", "GL_ARB_map_buffer_range", "GL_OES_EGL_image", "GL_ARB_half_float_vertex", "GL_ARB_sync", "GL_EXT_vertex_array_bgra", "GL_ARB_sampler_objects", "GL_EXT_framebuffer_multisample", "GL_ARB_texture_buffer_range", "GL_ARB_texture_storage", "GL_ARB_seamless_cube_map", "GL_ARB_framebuffer_object", "GL_ARB_program_interface_query", "GL_ARB_draw_buffers_blend", "GL_MESA_texture_signed_rgba", "GL_EXT_texture_snorm", "GL_ARB_texture_float", "GL_ARB_compressed_texture_pixel_storage", "GL_ARB_vertex_shader", "GL_ARB_vertex_type_2_10_10_10_rev", "GL_ARB_depth_clamp", "GL_EXT_texture_sRGB_decode", "GL_ARB_texture_multisample", "GL_AMD_vertex_shader_viewport_index", "GL_ARB_texture_rgb10_a2ui", "GL_NV_packed_depth_stencil", "GL_ARB_texture_buffer_object_rgb32", "GL_ARB_ES3_compatibility", "GL_NV_conditional_render", "GL_ARB_debug_output", "GL_AMD_seamless_cubemap_per_texture", "GL_EXT_framebuffer_blit", "GL_ARB_pipeline_statistics_query", "GL_ARB_shading_language_packing", "GL_MESA_ycbcr_texture", "GL_ARB_blend_func_extended", "GL_ARB_draw_indirect", "GL_ARB_invalidate_subdata", "GL_EXT_draw_buffers2", "GL_ARB_texture_mirror_clamp_to_edge", "GL_ARB_fragment_coord_conventions", "GL_IBM_multimode_draw_arrays", "GL_ANGLE_texture_compression_dxt3", "GL_ARB_shader_texture_lod", "GL_ARB_texture_stencil8", "GL_ARB_transform_feedback_instanced", "GL_ARB_vertex_array_object", "GL_ARB_gpu_shader_fp64", "GL_AMD_conservative_depth", "GL_ARB_texture_query_levels", "GL_ARB_transform_feedback2", "GL_ARB_explicit_attrib_location", "GL_ARB_draw_buffers", "GL_EXT_framebuffer_sRGB", "GL_ARB_shader_bit_encoding", "GL_ARB_vertex_type_10f_11f_11f_rev", "GL_AMD_draw_buffers_blend", "GL_ARB_texture_cube_map_array", "GL_EXT_transform_feedback", "GL_ARB_explicit_uniform_location", "GL_EXT_timer_query", "GL_ARB_internalformat_query2", "GL_EXT_texture_swizzle", "GL_MESA_pack_invert", "GL_ARB_texture_swizzle", "GL_EXT_texture_compression_dxt1", "GL_ARB_ES2_compatibility", "GL_ARB_clip_control", "GL_ARB_conservative_depth", "GL_ARB_direct_state_access", "GL_ARB_polygon_offset_clamp", "GL_EXT_blend_equation_separate", "GL_EXT_pixel_buffer_object", "GL_ARB_shader_subroutine", "GL_ARB_vertex_attrib_binding", "GL_ARB_half_float_pixel", "GL_ARB_instanced_arrays", "GL_ARB_occlusion_query2", "GL_ARB_texture_buffer_object", "GL_ARB_arrays_of_arrays", "GL_ARB_texture_gather", "GL_ARB_multi_draw_indirect", "GL_EXT_texture_shared_exponent", "GL_ATI_texture_mirror_once", "GL_EXT_texture_compression_s3tc", "GL_EXT_framebuffer_multisample_blit_scaled", "GL_ARB_clear_texture", "GL_ARB_texture_storage_multisample", "GL_EXT_texture_integer"} }

... lots of (probably) non-useful messages removed ...

[amethyst_assets::storage][DEBUG] "renderer::Texture": Asset "/home/gitlab-runner/builds/28120612/1/azriel91/autexousious/assets/test/object/character/bat/bat_grey.png" (handle id: Handle { id: 7, marker: PhantomData }) has been loaded successfully
[game_play::character_entity_spawner][DEBUG] Retrieving character with handle: `Handle { id: 0, marker: PhantomData }`
[amethyst::app][INFO] Engine is shutting down
test character_entity_spawner::test::spawn_for_player_creates_entity_with_object_components ... ok

test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

error: process didn't exit successfully: `/home/gitlab-runner/builds/28120612/1/azriel91/autexousious/target/debug/deps/game_play-482ef6f20580cf49` (signal: 11, SIGSEGV: invalid memory reference)
Run under strace (CI user) 
strace /home/gitlab-runner/builds/28120612/1/azriel91/autexousious/target/debug/deps/game_play-482ef6f20580cf49
... lots of stuff ...
[game_play::character_entity_spawner][DEBUG] Retrieving character with handle: `Handle { id: 0, marker: PhantomData }`
[amethyst::app][INFO] Engine is shutting down
) = 0
futex(0x7f911b822000, FUTEX_WAKE_PRIVATE, 1) = 0
write(1, "test character_entity_spawner::t"..., 96test character_entity_spawner::test::spawn_for_player_creates_entity_with_object_components ... ) = 96
write(1, "\33[32mok\33(B\33[m", 13ok)     = 13
write(1, "\n", 1
)                       = 1
write(1, "\n", 1
)                       = 1
write(1, "test result: ", 13test result: )           = 13
write(1, "\33[32mok\33(B\33[m", 13ok)     = 13
write(1, ". 2 passed; 0 failed; 0 ignored;"..., 61. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

) = 61
sigaltstack({ss_sp=NULL, ss_flags=SS_DISABLE, ss_size=8192}, NULL) = 0
munmap(0x7f911da84000, 8192)            = 0
futex(0x7f911ba90970, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable)
futex(0x7f911ba90970, FUTEX_WAIT_PRIVATE, 2, NULL) = ?
+++ killed by SIGSEGV (core dumped) +++
Segmentation fault (core dumped)
Run under rust-gdb (CI user) 
(gdb) run
... lots of stuff ...
[game_play::character_entity_spawner][DEBUG] Retrieving character with handle: `Handle { id: 0, marker: PhantomData }`
[amethyst::app][INFO] Engine is shutting down
test character_entity_spawner::test::spawn_for_player_creates_entity_with_object_components ... ok

test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

[Thread 0x7ffff4aae700 (LWP 24715) exited]
[Thread 0x7ffff33fe700 (LWP 24717) exited]
[Thread 0x7ffff43ff700 (LWP 24716) exited]
[Thread 0x7ffff41fe700 (LWP 24713) exited]
[Thread 0x7ffff3bff700 (LWP 24712) exited]
[Thread 0x7ffff35ff700 (LWP 24711) exited]
[Thread 0x7ffff31fd700 (LWP 24710) exited]
[Thread 0x7ffff5bfe700 (LWP 24714) exited]
[Thread 0x7fffe4b82700 (LWP 24709) exited]
[Thread 0x7fffe5383700 (LWP 24708) exited]
[Thread 0x7fffe5b84700 (LWP 24707) exited]
[Thread 0x7fffe6385700 (LWP 24706) exited]
[Thread 0x7fffe6b86700 (LWP 24705) exited]
[Thread 0x7fffe7387700 (LWP 24704) exited]
[Thread 0x7fffe7b88700 (LWP 24703) exited]
[Thread 0x7fffe8389700 (LWP 24702) exited]

Thread 1 "game_play-482ef" received signal SIGSEGV, Segmentation fault.
0x0000555556013853 in je_arena_bitselm_get_mutable (chunk=0x7fffe3800000, pageind=<optimized out>) at /checkout/src/liballoc_jemalloc/../jemalloc/include/jemalloc/internal/arena.h:710
710	/checkout/src/liballoc_jemalloc/../jemalloc/include/jemalloc/internal/arena.h: No such file or directory.
(gdb) info stack
#0  0x0000555556013853 in je_arena_bitselm_get_mutable (chunk=0x7fffe3800000, pageind=<optimized out>) at /checkout/src/liballoc_jemalloc/../jemalloc/include/jemalloc/internal/arena.h:710
#1  je_arena_mapbitsp_get_mutable (chunk=0x7fffe3800000, pageind=<optimized out>) at /checkout/src/liballoc_jemalloc/../jemalloc/include/jemalloc/internal/arena.h:788
#2  je_arena_mapbitsp_get_const (chunk=0x7fffe3800000, pageind=<optimized out>) at /checkout/src/liballoc_jemalloc/../jemalloc/include/jemalloc/internal/arena.h:795
#3  je_arena_mapbits_get (chunk=0x7fffe3800000, pageind=<optimized out>) at /checkout/src/liballoc_jemalloc/../jemalloc/include/jemalloc/internal/arena.h:809
#4  arena_run_reg_dalloc (run=<optimized out>, ptr=<optimized out>) at /checkout/src/liballoc_jemalloc/../jemalloc/src/arena.c:306
#5  arena_dalloc_bin_locked_impl (tsdn=<optimized out>, arena=<optimized out>, chunk=<optimized out>, ptr=<optimized out>, bitselm=<optimized out>, junked=<optimized out>)
    at /checkout/src/liballoc_jemalloc/../jemalloc/src/arena.c:2963
#6  0x000055555602aa87 in je_tcache_bin_flush_small (tsd=<optimized out>, tcache=<optimized out>, tbin=0x7ffff5e0d168, binind=<optimized out>, rem=64) at /checkout/src/liballoc_jemalloc/../jemalloc/src/tcache.c:137
#7  0x000055555600905c in je_tcache_dalloc_small (tsd=0x7fffe3800000, tcache=<optimized out>, ptr=<optimized out>, binind=10, slow_path=false)
    at /checkout/src/liballoc_jemalloc/../jemalloc/include/jemalloc/internal/tcache.h:419
#8  je_arena_dalloc (tsdn=<optimized out>, ptr=<optimized out>, tcache=<optimized out>, slow_path=false) at /checkout/src/liballoc_jemalloc/../jemalloc/include/jemalloc/internal/arena.h:1441
#9  je_idalloctm (is_metadata=false, slow_path=false, tsdn=<optimized out>, ptr=<optimized out>, tcache=<optimized out>) at include/jemalloc/internal/jemalloc_internal.h:1170
#10 je_iqalloc (tsd=<optimized out>, ptr=<optimized out>, tcache=<optimized out>, slow_path=<optimized out>) at include/jemalloc/internal/jemalloc_internal.h:1187
#11 ifree (tsd=<optimized out>, ptr=<optimized out>, tcache=<optimized out>, slow_path=<optimized out>) at /checkout/src/liballoc_jemalloc/../jemalloc/src/jemalloc.c:1896
#12 0x00007fffed395a3a in ?? () from /usr/lib/x86_64-linux-gnu/dri/swrast_dri.so
#13 0x00007fffed3421fc in ?? () from /usr/lib/x86_64-linux-gnu/dri/swrast_dri.so
#14 0x00007fffed373569 in ?? () from /usr/lib/x86_64-linux-gnu/dri/swrast_dri.so
#15 0x00007fffed107929 in ?? () from /usr/lib/x86_64-linux-gnu/dri/swrast_dri.so
#16 0x00007ffff6cb6041 in __run_exit_handlers (status=0, listp=0x7ffff705e718 <__exit_funcs>, run_list_atexit=run_list_atexit@entry=true, run_dtors=run_dtors@entry=true) at exit.c:108
#17 0x00007ffff6cb613a in __GI_exit (status=<optimized out>) at exit.c:139
#18 0x00007ffff6c94b9e in __libc_start_main (main=0x5555556028e0 <main>, argc=1, argv=0x7fffffffe1f8, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fffffffe1e8)
    at ../csu/libc-start.c:344
#19 0x000055555559700a in _start ()

I included the GL information in the first two in case that contains any clues, I'm a complete rookie at gdb, so I don't yet know how to interpret that stack info, except it looks like it's trying to free memory where it's not meant to.

Is it segfaulting during teardown of the library test, before the doc test? Is it possible to tell if it's due to a fault in my code (I have no unsafe code), rustc, or some system library?

2

Ouch. Can you share Cargo.lock?

(Initially I requested just the dependencies list because I thought this was CI running on a clean system, but I'm not sure what your setup is here. What I'm interested in seeing is the versions of packages actually used in the bad build of your crate)

Based on the output, this superficially looks like a double-free error (or freeing a null pointer?) in an external library... maybe? but segfaults usually have layers upon layers of surprises hiding beneath them. So it's often easier IMO to start from the source (in this case, of your dependencies) and look for the possible causes of unsafety there.

Is it segfaulting during teardown of the library test, before the doc test?

Doc-tests and library tests are separate executables, so you verified it was the library test when you ran the executable directly.

1 Like
4

On your user system could try running with;
env LIBGL_ALWAYS_SOFTWARE=1
With any luck this will give the segfault by using the same GL library. (swrast_dri.so)

There is too little known to explicitly blame a single source. Guessing it could even possibly be an allocation with a different allocator.

I would personally if getting nowhere, try stripping out code to create a minimal test that triggers the error.

5

Ouch. Can you share Cargo.lock?

Yeap, here we go: Cargo.lock GL Segfault - Pastebin.com (discourse uploads has a whitelist for image files)

On your user system could try running with;
env LIBGL_ALWAYS_SOFTWARE=1
With any luck this will give the segfault by using the same GL library. (swrast_dri.so)

Oh hey this made the segfault happen. Yay - kind of (haha)

I would personally if getting nowhere, try stripping out code to create a minimal test that triggers the error

Shall try!

6

it’s often easier IMO to start from the source (in this case, of your dependencies) and look for the possible causes of unsafety there.

Sadly I couldn't find anything by going into the GL-related repos and doing quick searches through the issues / code.

create a minimal test that triggers the error

Here we go:

git clone git@github.com:azriel91/amethyst_gl_segfault.git
cd amethyst_gl_segfault
export LIBGL_ALWAYS_SOFTWARE=1 LIBGL_DEBUG=verbose
cargo test

That gives me the segfault:

$ cargo test
   Compiling amethyst_gl_segfault v0.1.0 (file:///home/azriel/work/github/azriel91/amethyst_gl_segfault)
    Finished dev [unoptimized + debuginfo] target(s) in 7.29s
     Running target/debug/deps/amethyst_gl_segfault-8e9c5ca4ba7c6deb

running 1 test
libGL: OpenDriver: trying /usr/lib/x86_64-linux-gnu/dri/tls/swrast_dri.so
libGL: OpenDriver: trying /usr/lib/x86_64-linux-gnu/dri/swrast_dri.so
libGL: Can't open configuration file /home/azriel/.drirc: No such file or directory.
libGL: Can't open configuration file /home/azriel/.drirc: No such file or directory.
[amethyst::app][INFO] Initializing Amethyst...
[amethyst::app][INFO] Version: 0.7.0
[amethyst::app][INFO] Platform: x86_64-unknown-linux-gnu
[amethyst::app][INFO] Amethyst git commit: 00fa904612be5609121981a9663a7aea097e3a55
[amethyst::app][INFO] Rustc version: 1.28.0-nightly Nightly
[amethyst::app][INFO] Rustc git commit: b68432d560c7c6f1e738b27e49d271a2a778f898
[amethyst_assets::storage][DEBUG] "renderer::Texture": Asset "<Data>" (handle id: Handle { id: 0, marker: PhantomData }) has been loaded successfully
[amethyst_assets::storage][DEBUG] "renderer::Texture": Asset "<Data>" (handle id: Handle { id: 1, marker: PhantomData }) has been loaded successfully
[amethyst_assets::storage][DEBUG] "renderer::Texture": Asset "<Data>" (handle id: Handle { id: 2, marker: PhantomData }) has been loaded successfully
[amethyst_assets::storage][DEBUG] "renderer::Texture": Asset "<Data>" (handle id: Handle { id: 3, marker: PhantomData }) has been loaded successfully
[amethyst_assets::storage][DEBUG] "renderer::Texture": Asset "<Data>" (handle id: Handle { id: 4, marker: PhantomData }) has been loaded successfully
[amethyst_assets::storage][DEBUG] "renderer::Texture": Asset "<Data>" (handle id: Handle { id: 5, marker: PhantomData }) has been loaded successfully
[amethyst_assets::storage][DEBUG] "renderer::Texture": Asset "<Data>" (handle id: Handle { id: 6, marker: PhantomData }) has been loaded successfully
[amethyst::app][INFO] Engine is shutting down
test tests::segfault ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

error: process didn't exit successfully: `/home/azriel/work/github/azriel91/amethyst_gl_segfault/target/debug/deps/amethyst_gl_segfault-8e9c5ca4ba7c6deb` (signal: 11, SIGSEGV: invalid memory reference)
7

Does this segfault after the application quit? That means it's probably some thread that outlives the Application..

8

That seems both correct and strange. The segfault doesn't happen if the same application code is compiled as a binary without the #[test] bits around it (see the run-as-application branch if you're interested).

Information gathered so far:

  • The segfault happens on Ubuntu 18.04 when using software rendering:
    export LIBGL_ALWAYS_SOFTWARE=1
  • It doesn't happen on Windows; OS X fails on a different error (can't run UI on non-main thread)
  • It doesn't happen when using hardware rendering.
  • It doesn't happen when the same application is run outside of the test context.
  • It doesn't always happen — I've run the test binary multiple times and have seen it not segfault sometimes. It usually happens though.
1 Like
9

Here is my interpretation of your observations so far:

  • The OpenGL implementation in use obviously plays a key role here. It's too early to tell if it is the culprit though. Maybe it does something wrong, maybe it is used incorrectly in a way which will crash the specific algorithm in use.
  • If it is intermittent and disturbs the application non-locally, it is very likely to be an memory safety issue (lifetime issue, double-free, data race, buffer overflow...).
  • One key difference between cargo test and a "regular" main is that cargo test runs your tests in a separate thread. We may want to investigate this further, because...
  • The Linux 3D stack is known to have some serious thread-safety issues (just go and have a look at KDE's bugtracker, they recently decided to make their renderer multi-threaded and have a lot of issues with Mesa drivers now).

Can you try to run your test binary in valgrind's memcheck and helgrind to see if they find anything interesting? For best results, you will want to install debug information for the libraries that you use, especially your software rendering implementation of course.

Another test which you may want to perform is to run cargo test in single-threaded mode ("--test-threads=1"), but I'm not sure if this will prevent cargo test from firing up a separate thread for the test harness. Conversely, you may want to modify your freestanding application so that it runs everything in a separate thread, just to see it changes anything.

4 Likes
10

Gave it a go. Same segfault.
Can reproduce with a binary

fn main() {
    use std::sync::mpsc::channel;
    use std::thread;
    let (sender, receiver) = channel();
    let _t = thread::spawn(move || {
        assert!(run().is_ok());
        sender.send(()).unwrap();
    });
    receiver.recv().unwrap();
}

Crash can be on either thread.
Mesa will be missing some synchronization when both close at same time. Not installed symbols so stack not useful.

Easy workaround would be just to have your test function run the code in a sub-thread and join it.

4 Likes