Reverse Engineering μ-architectures

See INSTALL.md for installation instructions

Cache coherence protocols

• Question about Intel's cache coherence
• RSP runs on a ARM Cortex-A72
  • According to ARM's infocenter this chip uses the MESI protocol for its L1 cache
• More information on Intel's cache hierarchy
• Cache coherence microbenchmarks
• Cache coherence implementation notes
• Cache coherence protocols evaluation
• Info on ARM Cortex Coherence
• Microbenchmarking GPU Memory architecture
• Demystifying GPU Memory Hierarchy
• Intel’s CLDEMOTE instruction
• Intel's HitME cache

Milestones

• (31st October): Choose range of platforms to verify end system on
  • (1 ARM, 2 Intel):
• (31st October): Choose main investigative architecture (RSP, BBB, Intel Core-i5)
  • Personal Linux laptop: Intel(R) Core(TM) i5-2520M CPU @ 2.50GHz
  • Condor server: 24 Core Intel(R) Xeon(R) Platinum 8275CL CPU @ 3.00GHz
    • Architecture overview
  • RSP Model 4B
    • Quad core Cortex-A72 (ARM v8) 64-bit SoC @ 1.5GHz
    • Some slides on RSP cache
• (12th November): Devise which μ-architectural features we want to extract
  • Intel's Smart Cache outlines useful cache design choices
  • State machine: does cache behaviour conform to MESIF/MESI? How can the benchmark be useful while assuming an incorrect protocol?
  • Replacement policies: LRU? Custom?
  • How does the coherence protocol change with numbers of cores? (4 on RPI vs. 22 on server)
  • How does the coherence protocol change with OS? (Linux vs. FreeBSD)
  • Implementation: Directory-based, snooping, invalidation-based, update-based, write-allocation, bus-system, inclusion policy, lock-up free etc.
  • Smart cache:
    • "Intel Smart Cache also features a new power-saving mechanism that enables the L2 Intel Smart Cache to dynamically flush its ways into system memory, based on demand, or during periods of inactivity"
      • Can we test this?
  • Are there any optimizations for common cache access patterns, e.g., false sharing, stampede, deadlocks, data contention etc.
  • Are there optimizations for excessive amounts of consecutive misses?
  • Are there optimizations for misses that come in a pattern. E.g., H M M M H M M M H ...
• (12th November): Devise which microbenchmarks could uncover above features
• (12th November): Write initial benchmarks and write measurement/benchmark infrastructure (likely using perf)
• (17th November):
  • Able to set coherence state programmatically
  • Benchmark suite for profiling coherence information

Notes

• Can we take inspiration from Fuzzers? E.g., CSmith
• Can we take insipration from multi-core litmus tests? E.g., Herd

CPU Info

Server

$> lscpu

Architecture:        x86_64
CPU op-mode(s):      32-bit, 64-bit
Byte Order:          Little Endian
CPU(s):              96
On-line CPU(s) list: 0-95
Thread(s) per core:  2
Core(s) per socket:  24
Socket(s):           2
NUMA node(s):        2
Vendor ID:           GenuineIntel
CPU family:          6
Model:               85
Model name:          Intel(R) Xeon(R) Platinum 8275CL CPU @ 3.00GHz
Stepping:            7
CPU MHz:             1200.021
CPU max MHz:         3900.0000
CPU min MHz:         1200.0000
BogoMIPS:            6000.00
Virtualization:      VT-x
L1d cache:           32K
L1i cache:           32K
L2 cache:            1024K
L3 cache:            36608K
NUMA node0 CPU(s):   0-23,48-71
NUMA node1 CPU(s):   24-47,72-95
Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 invpcid_single ssbd mba ibrs ibpb stibp ibrs_enhanced tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb intel_pt avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req pku ospke avx512_vnni md_clear flush_l1d arch_capabilities

Flag Index

Legend can be found here

• clflush: supports cache line flush instruction
• dca: direct cache access
• cat_l3: Cache Allocation Technology L3
• cdp_l3: Code and Data Prioritization L3
• cqm: Cache QoS Monitoring
• cqm_llc: LLC QoS
• cqm_occup_llc: LLC occupancy monitoring
• cqm_mbm_total: LLC total MBM monitoring
• cqm_mbm_local: LLC local MBM monitoring

Implications

This processor is based on the Cascade Lake micro-architecture (see manual).

According to the IA64 optimization manual section 7.3.3.3 Cascade Lake processor with 2 sockets contains two CL advanced performance packages where each package is made of two processor dies connected via a Intel Ultra Path Interconnect creating four NUMA domains

This architecture has a larger than usual L2 and L3 cache and its L1 cache is split into data and instruction cache of 32KB each. The processor has 2 sockets with 24 cores each. In the Intel Xeon architecture this implies that

Tools

• Emon
• Herd

TODO

• Parameterize benchmarks
• Benchmark result aggregation
• Measure from multiple cores
• Measure LLC - L2 on core
• Graphs
• Profile and quantify framework overheads:
  • Loop overheads
  • Scheduler synchronization/core pinning overheads
  • rdtsc noise
  • Cold start/warm start (including for state initialization)
  • Turbo-boost
  • Other HW/OS/compiler optimizations
• Validate state initialization logic
• Add pmc and libpfm support
• Detecting protocol
  • Protocol transition map
• Run on other platforms (ARM, AMD, more Intel micro-architectures)
• Simplify/extend utility scripts
• Measurement threads
• Convert rdtsc return to time unit (instead of cycles)

Branch predictors

• ARM Cortex M3 Pipeline Forum Question
• Arduino ARM Cortex-A8
• ARM Cortex A8 Branch Predictor Infocenter
• Possible cheap Intel boards:
  • Arduino Intel Galileo Gen2
  • Arduino Intel Edison
    • Intel Atom's CPU (used in Intel Edison) does have a branch predictor as detailed in this manual
    • Alternatively, could use the Intel Xeon Platinum processor as on the condor server because of its lack of documentation
    • Possibly could microbenchmark generation framework similar to this