mersenneforum.org  

Go Back   mersenneforum.org > Great Internet Mersenne Prime Search > Hardware > GPU Computing

Reply
 
Thread Tools
Old 2016-05-21, 14:12   #34
bgbeuning
 
Dec 2014

3·5·17 Posts
Default

I am a software guy and not a hardware guy. But after the IO port conflict I
started wondering about getting to the 8 th GPU also.

I switched the disk from AHCI to IDE but it seems to be using more IO ports.

Quote:
0d00-ffff : PCI Bus 0000:00
1000-1fff : PCI Bus 0000:01
1000-101f : 0000:01:00.0
1020-103f : 0000:01:00.1
2880-288f : 0000:00:1f.5
2880-288f : ata_piix
2c00-2c0f : 0000:00:1f.5
2c00-2c0f : ata_piix
3000-3003 : 0000:00:1f.5
3000-3003 : ata_piix
3080-3087 : 0000:00:1f.5
3080-3087 : ata_piix
3400-3403 : 0000:00:1f.5
3400-3403 : ata_piix
3480-3487 : 0000:00:1f.5
3480-3487 : ata_piix
3880-388f : 0000:00:1f.2
3880-388f : ata_piix
3c00-3c0f : 0000:00:1f.2
3c00-3c0f : ata_piix
4000-4003 : 0000:00:1f.2
4000-4003 : ata_piix
4080-4087 : 0000:00:1f.2
4080-4087 : ata_piix
4400-4403 : 0000:00:1f.2
4400-4403 : ata_piix
4480-4487 : 0000:00:1f.2
4480-4487 : ata_piix
4800-481f : 0000:00:1d.2
4800-481f : uhci_hcd
4880-489f : 0000:00:1d.1
4880-489f : uhci_hcd
4c00-4c1f : 0000:00:1d.0
4c00-4c1f : uhci_hcd
5000-5fff : PCI Bus 0000:1a
5000-507f : 0000:1a:04.0
Wanted to disable the serial ports but did not find that in the BIOS.
If I disable USB will the keyboard still work for the BIOS?
Some C6100 use PS/2 keyboard and mouse, but mine uses USB.

Quote:
bgb@c6100a1:~$ lspci -s 1a:04.0
1a:04.0 VGA compatible controller: ASPEED Technology, Inc. ASPEED Graphics Family (rev 10)
All of the echo 1 failed

Quote:
bgb@c6100a1:/sys/bus/pci/devices/0000:10:00.0$ sudo echo 1 > remove
-bash: remove: Permission denied
bgb@c6100a1:/sys/bus/pci/devices/0000:10:00.0$ sudo echo 1 > rescan
-bash: rescan: Permission denied
The kernel sees all 8 GPU but does not make them available.

Quote:
bgb@c6100a1:~$ lspci | grep -i nvidia
04:00.0 PCI bridge: NVIDIA Corporation NF200 PCIe 2.0 switch for Quadro Plex S4 / Tesla S870 / Tesla S1070 / Tesla S2050 (rev a3)
05:00.0 PCI bridge: NVIDIA Corporation NF200 PCIe 2.0 switch for Quadro Plex S4 / Tesla S870 / Tesla S1070 / Tesla S2050 (rev a3)
05:01.0 PCI bridge: NVIDIA Corporation NF200 PCIe 2.0 switch for Quadro Plex S4 / Tesla S870 / Tesla S1070 / Tesla S2050 (rev a3)
05:02.0 PCI bridge: NVIDIA Corporation NF200 PCIe 2.0 switch for Quadro Plex S4 / Tesla S870 / Tesla S1070 / Tesla S2050 (rev a3)
05:03.0 PCI bridge: NVIDIA Corporation NF200 PCIe 2.0 switch for Quadro Plex S4 / Tesla S870 / Tesla S1070 / Tesla S2050 (rev a3)
0f:00.0 3D controller: NVIDIA Corporation GF100GL [Tesla M2070] (rev a3)
0f:00.1 Audio device: NVIDIA Corporation GF100 High Definition Audio Controller (rev a1)
10:00.0 3D controller: NVIDIA Corporation GF100GL [Tesla M2070] (rev a3)
10:00.1 Audio device: NVIDIA Corporation GF100 High Definition Audio Controller (rev a1)
11:00.0 3D controller: NVIDIA Corporation GF100GL [Tesla M2070] (rev a3)
11:00.1 Audio device: NVIDIA Corporation GF100 High Definition Audio Controller (rev a1)
12:00.0 3D controller: NVIDIA Corporation GF100GL [Tesla M2070] (rev a3)
12:00.1 Audio device: NVIDIA Corporation GF100 High Definition Audio Controller (rev a1)
15:00.0 3D controller: NVIDIA Corporation GF100GL [Tesla M2070] (rev a3)
15:00.1 Audio device: NVIDIA Corporation GF100 High Definition Audio Controller (rev a1)
16:00.0 3D controller: NVIDIA Corporation GF100GL [Tesla M2070] (rev a3)
16:00.1 Audio device: NVIDIA Corporation GF100 High Definition Audio Controller (rev a1)
17:00.0 3D controller: NVIDIA Corporation GF100GL [Tesla M2070] (rev a3)
17:00.1 Audio device: NVIDIA Corporation GF100 High Definition Audio Controller (rev a1)
18:00.0 3D controller: NVIDIA Corporation GF100GL [Tesla M2070] (rev a3)
18:00.1 Audio device: NVIDIA Corporation GF100 High Definition Audio Controller (rev a1)
Tell me if I missed something you asked for.
bgbeuning is offline   Reply With Quote
Old 2016-05-21, 14:14   #35
bgbeuning
 
Dec 2014

3·5·17 Posts
Default

Quote:
Originally Posted by tentonine View Post
One other thing - please can you send the output from:
lspci -tv
It might be necessary to attach it as a text file. It might help for me to see what is going on with your BAR errors a little more clearly.

Thanks.
I supposed I could google it, but what is a BAR?

Quote:
bgb@c6100a1:~$ lspci -tv
-+-[0000:ff]-+-00.0 Intel Corporation Xeon 5500/Core i7 QuickPath Architecture Generic Non-Core Registers
| +-00.1 Intel Corporation Xeon 5500/Core i7 QuickPath Architecture System Address Decoder
| +-02.0 Intel Corporation Xeon 5500/Core i7 QPI Link 0
| +-02.1 Intel Corporation Xeon 5500/Core i7 QPI Physical 0
| +-02.4 Intel Corporation Xeon 5500/Core i7 QPI Link 1
| +-02.5 Intel Corporation Xeon 5500/Core i7 QPI Physical 1
| +-03.0 Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller
| +-03.1 Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Target Address Decoder
| +-03.2 Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller RAS Registers
| +-03.4 Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Test Registers
| +-04.0 Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 0 Control Registers
| +-04.1 Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 0 Address Registers
| +-04.2 Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 0 Rank Registers
| +-04.3 Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 0 Thermal Control Registers
| +-05.0 Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 1 Control Registers
| +-05.1 Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 1 Address Registers
| +-05.2 Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 1 Rank Registers
| +-05.3 Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 1 Thermal Control Registers
| +-06.0 Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 2 Control Registers
| +-06.1 Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 2 Address Registers
| +-06.2 Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 2 Rank Registers
| \-06.3 Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 2 Thermal Control Registers
+-[0000:fe]-+-00.0 Intel Corporation Xeon 5500/Core i7 QuickPath Architecture Generic Non-Core Registers
| +-00.1 Intel Corporation Xeon 5500/Core i7 QuickPath Architecture System Address Decoder
| +-02.0 Intel Corporation Xeon 5500/Core i7 QPI Link 0
| +-02.1 Intel Corporation Xeon 5500/Core i7 QPI Physical 0
| +-02.4 Intel Corporation Xeon 5500/Core i7 QPI Link 1
| +-02.5 Intel Corporation Xeon 5500/Core i7 QPI Physical 1
| +-03.0 Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller
| +-03.1 Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Target Address Decoder
| +-03.2 Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller RAS Registers
| +-03.4 Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Test Registers
| +-04.0 Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 0 Control Registers
| +-04.1 Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 0 Address Registers
| +-04.2 Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 0 Rank Registers
| +-04.3 Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 0 Thermal Control Registers
| +-05.0 Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 1 Control Registers
| +-05.1 Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 1 Address Registers
| +-05.2 Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 1 Rank Registers
| +-05.3 Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 1 Thermal Control Registers
| +-06.0 Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 2 Control Registers
| +-06.1 Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 2 Address Registers
| +-06.2 Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 2 Rank Registers
| \-06.3 Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 2 Thermal Control Registers
\-[0000:00]-+-00.0 Intel Corporation 5520 I/O Hub to ESI Port
+-01.0-[01]--+-00.0 Intel Corporation 82576 Gigabit Network Connection
| \-00.1 Intel Corporation 82576 Gigabit Network Connection
+-03.0-[02]--
+-05.0-[03]--
+-07.0-[04-19]----00.0-[05-19]--+-00.0-[06]--
| +-01.0-[07]--
| +-02.0-[08-18]----00.0-[09-18]--+-04.0-[0a-12]----00.0-[0b-12]--+-00.0-[0c]--
| | | \-04.0-[0d-12]----00.0-[0e-12]--+-04.0-[0f]--+-00.0 NVIDIA Corporation GF100GL [Tesla M2070]
| | | | \-00.1 NVIDIA Corporation GF100 High Definition Audio Controller
| | | +-08.0-[10]--+-00.0 NVIDIA Corporation GF100GL [Tesla M2070]
| | | | \-00.1 NVIDIA Corporation GF100 High Definition Audio Controller
| | | +-10.0-[11]--+-00.0 NVIDIA Corporation GF100GL [Tesla M2070]
| | | | \-00.1 NVIDIA Corporation GF100 High Definition Audio Controller
| | | \-14.0-[12]--+-00.0 NVIDIA Corporation GF100GL [Tesla M2070]
| | | \-00.1 NVIDIA Corporation GF100 High Definition Audio Controller
| | \-08.0-[13-18]----00.0-[14-18]--+-04.0-[15]--+-00.0 NVIDIA Corporation GF100GL [Tesla M2070]
| | | \-00.1 NVIDIA Corporation GF100 High Definition Audio Controller
| | +-08.0-[16]--+-00.0 NVIDIA Corporation GF100GL [Tesla M2070]
| | | \-00.1 NVIDIA Corporation GF100 High Definition Audio Controller
| | +-10.0-[17]--+-00.0 NVIDIA Corporation GF100GL [Tesla M2070]
| | | \-00.1 NVIDIA Corporation GF100 High Definition Audio Controller
| | \-14.0-[18]--+-00.0 NVIDIA Corporation GF100GL [Tesla M2070]
| | \-00.1 NVIDIA Corporation GF100 High Definition Audio Controller
| \-03.0-[19]--
+-13.0 Intel Corporation 7500/5520/5500/X58 I/O Hub I/OxAPIC Interrupt Controller
+-14.0 Intel Corporation 7500/5520/5500/X58 I/O Hub System Management Registers
+-14.1 Intel Corporation 7500/5520/5500/X58 I/O Hub GPIO and Scratch Pad Registers
+-14.2 Intel Corporation 7500/5520/5500/X58 I/O Hub Control Status and RAS Registers
+-14.3 Intel Corporation 7500/5520/5500/X58 I/O Hub Throttle Registers
+-16.0 Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device
+-16.1 Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device
+-16.2 Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device
+-16.3 Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device
+-16.4 Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device
+-16.5 Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device
+-16.6 Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device
+-16.7 Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device
+-1d.0 Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #1
+-1d.1 Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #2
+-1d.2 Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #3
+-1d.7 Intel Corporation 82801JI (ICH10 Family) USB2 EHCI Controller #1
+-1e.0-[1a]----04.0 ASPEED Technology, Inc. ASPEED Graphics Family
+-1f.0 Intel Corporation 82801JIR (ICH10R) LPC Interface Controller
+-1f.2 Intel Corporation 82801JI (ICH10 Family) 4 port SATA IDE Controller #1
+-1f.3 Intel Corporation 82801JI (ICH10 Family) SMBus Controller
\-1f.5 Intel Corporation 82801JI (ICH10 Family) 2 port SATA IDE Controller #2
bgb@c6100a1:~$ ^C
bgb@c6100a1:~$
bgbeuning is offline   Reply With Quote
Old 2016-05-21, 14:34   #36
Mark Rose
 
Mark Rose's Avatar
 
"/X\(‘-‘)/X\"
Jan 2013

23·32·43 Posts
Default

Quote:
Originally Posted by bgbeuning View Post
All of the echo 1 failed
That's because sudo doesn't apply to redirected input/output. Either run `sudo -i` to get an interactive root shell and run those again, or use the `tee` command:

echo 1 | sudo tee /sys/bus/pci/devices/0000:10:00.0/remove

Last fiddled with by Mark Rose on 2016-05-21 at 14:34
Mark Rose is offline   Reply With Quote
Old 2016-05-21, 15:05   #37
bgbeuning
 
Dec 2014

3×5×17 Posts
Default

The ipmitool returned some errors.

The sticker on the top front of the case does say version A00 like yours.

Here is a screen shot of the web interface showing "8 to 1" configured.
Notice the picture says "Control by O Jumper X BMC" which leads me
to think there might be jumpers inside the case to configure "8 to 1" mode.
Attached Thumbnails
Click image for larger version

Name:	c410x.PNG
Views:	259
Size:	73.0 KB
ID:	14407  
bgbeuning is offline   Reply With Quote
Old 2016-05-21, 15:48   #38
bgbeuning
 
Dec 2014

3·5·17 Posts
Default

Quote:
Originally Posted by Mark Rose View Post
That's because sudo doesn't apply to redirected input/output.
Doh! Thanks Mark.

I tried a couple of the rescan and one remove.
nivida-smi gave no joy.
Removed and restarted the nvidia driver, and still no joy.
Since I do not know what I am doing here, I only tried a couple.
bgbeuning is offline   Reply With Quote
Old 2016-05-21, 21:23   #39
anonymous
 

32·229 Posts
Default

As for USB, I think the USB keyboard will stop working if you disable the ports. Do you actually need to use the server in person with a keyboard routinely and are you just concerned about getting back into the BIOS again later? If the only concern is the BIOS, perhaps a better option is to try disabling USB just in linux - disabling it in the BIOS will improve the chances of it working, as then the BIOS has an improved chance of assigning the memory correctly, but it is easier to try with Linux alone. To do this, press e to edit when at the grub menu and then add the following to the list of kernel parameters:
nousb
This takes effect for the current boot only, and will go away when you boot again. There is a way to make it permanent, if it helps.

If you were to disable USB in the BIOS, the answer to recovering keyboard input later would be to see if you can edit the BIOS settings within Linux (say via ssh connection) using Dell's setupbios tool, available here:
http://poweredgec.com/files/
And described here:
http://www.dell.com/support/article/us/en/04/SLN266215
I would suggest testing this first to make sure it works, before disabling USB and then needing to rely on it.

Another and better alternative (in case of any Linux problems occurring) is to set up serial console redirection via IPMI so that you can control the keyboard input remotely, but you would need to be very careful with setting this up, as there are apparently bugs with some firmware versions on the C410X that could cause problems and even brick your board, as described here: https://forums.servethehome.com/inde...oard-yet.1448/

In the worst case, you could unplug the power, pull out the node, remove the CMOS battery, and use the reset jumper on the motherboard, and that should reset the BIOS to get USB working again.


Thinking of firmware, have you checked your BIOS version on the C6100? An update might fix the problem. However, there is of course always some risk, as the board can be bricked if a problem occurs during the update. You apparently also have to be careful to ensure that you don't have a custom C6100 that doesn't work with the regular BIOS, as described here: http://www.copyerror.com/2013/08/10/...ll-c6100-bios/
  Reply With Quote
Old 2016-05-21, 21:25   #40
anonymous
 

2·4,703 Posts
Default

Sorry - I should have explained in more detail what I was talking about with the PCI stuff. A BAR is a base address register, i.e. a region of memory on the PCI card that needs to be mapped into the computer's memory or I/O ports. The dmesg output from earlier showed that one of the BARs was not being mapped correctly.

With the bus remove and subsequent rescan options I gave, I didn't explain clearly enough that they must be carried out in order: i.e. run the remove command, then the rescan that I placed on the next line - i.e. do not run the rescan on the device that you removed, but the bus device. Ideally, also run this rescan:
echo 1 > /sys/bus/pci/devices/0000:0e:04.0/pci_bus/<whatever>/rescan
where you will need to look to see what whatever is - there should only be a single entry there.
Essentially, the idea is to remove the GPU from the bus, then rescan the bus to try to pick it up again, where the bus in this case means a bridge chip in the C410X far down the tree, so that you don't end up rescanning every other GPU too.

Unfortunately the tree output isn't completely clear because the extra spacing was removed and so it is difficult to work out where the | lines should go to show the connections between bridges. Please could you attach the output from lspci -tv as a text file?

Please can you tell me what the error was with the ipmitool command and can you try again with the following - it might help:
ipmitool -I lanplus -H x.x.x.x -U root -P password fru print
Again, please replace password with your actual password and replace root with your actual username (if you changed it), and of course x.x.x.x with the IP address.
Just to confirm, was the IP definitely set correctly to the c410x's IP and can you reach the web browser interface on the C410X from the computer on which you can ran the command?

After looking at various boards I have seen for sale, they all only seem to have jumpers for switching between 2:1 and 4:1, so I think 8:1 is a software setting, making it difficult to determine which boards support it form visual inspection. I think that the chassis and board revisions need not be the same, so it would be really helpful for me to know if you have a Rev A00 board, A01, or even higher, along with the manufacturing date. I think you might have a Rev A01, even though the chassis is Rev A00.

Finally, just in case it is useful to you, did you know that straightforward control of the c410x is available from the command line, without needing to use the web browser? You can download C410x_Scripts.tar.gz and find inside a file called port_map.sh, which can be used to change the mapping from the command line. You can get this file from: http://poweredgec.com/latest_powered...stem Utilities (don't get a Scripts file version starting with an R - they don't support 8:1 mode).
Then there are also various power commands, including individual slot power commands, that you can see here:
http://www.dell.com/support/article/us/en/04/SLN244176
  Reply With Quote
Old 2016-05-22, 02:01   #41
bgbeuning
 
Dec 2014

3·5·17 Posts
Default

Here are the IPMI errors

Quote:
root@nas:/etc/dhcp# ipmitool -H 192.168.2.152 -U xxx -P xxx fru print
FRU Device Description : Builtin FRU Device (ID 0)
FRU Read failed: Cannot return number of requested data bytes
FRU Read failed: Cannot return number of requested data bytes
FRU Read failed: Cannot return number of requested data bytes

FRU Device Description : AST1100
FRU Read failed: Cannot return number of requested data bytes
FRU Read failed: Cannot return number of requested data bytes
FRU Read failed: Cannot return number of requested data bytes
Error code 0xCA prints this message.
bgbeuning is offline   Reply With Quote
Old 2016-05-22, 02:06   #42
bgbeuning
 
Dec 2014

3·5·17 Posts
Default

lspci -tv output
Attached Files
File Type: txt tv.txt (11.4 KB, 212 views)
bgbeuning is offline   Reply With Quote
Old 2016-05-22, 02:43   #43
bgbeuning
 
Dec 2014

3·5·17 Posts
Default

I opened up the box and took a picture of the PCB.
It has a sticker that says A00 but the PCB says "rev 1.1".
I will leave it open in case you want more pics.
Attached Thumbnails
Click image for larger version

Name:	Image21.jpg
Views:	204
Size:	636.4 KB
ID:	14412  
bgbeuning is offline   Reply With Quote
Old 2016-05-22, 02:54   #44
anonymous
 

7×17×79 Posts
Default

Thanks! That's very useful and kind of you to do that. Please can you also tell me the revision from the iPass connector cards, both on the sticker and on the PCB? In particular, the upper card is likely to be relevant.
  Reply With Quote
Reply

Thread Tools


Similar Threads
Thread Thread Starter Forum Replies Last Post
Running multiple ecm's johnadam74 GMP-ECM 21 2019-10-27 18:04
Multiple GPU's Windows 10 cardogab7341 GPU Computing 5 2015-08-09 13:57
Using multiple PCs numbercruncher Information & Answers 18 2014-04-17 00:17
Running multiple copies of mprime on Linux hc_grove Software 3 2004-10-10 15:34
Multiple systems/multiple CPUs. Best configuration? BillW Software 1 2003-01-21 20:11

All times are UTC. The time now is 14:24.


Fri Feb 3 14:24:20 UTC 2023 up 169 days, 11:52, 1 user, load averages: 1.00, 1.01, 0.99

Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2023, Jelsoft Enterprises Ltd.

This forum has received and complied with 0 (zero) government requests for information.

Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation.
A copy of the license is included in the FAQ.

≠ ± ∓ ÷ × · − √ ‰ ⊗ ⊕ ⊖ ⊘ ⊙ ≤ ≥ ≦ ≧ ≨ ≩ ≺ ≻ ≼ ≽ ⊏ ⊐ ⊑ ⊒ ² ³ °
∠ ∟ ° ≅ ~ ‖ ⟂ ⫛
≡ ≜ ≈ ∝ ∞ ≪ ≫ ⌊⌋ ⌈⌉ ∘ ∏ ∐ ∑ ∧ ∨ ∩ ∪ ⨀ ⊕ ⊗ 𝖕 𝖖 𝖗 ⊲ ⊳
∅ ∖ ∁ ↦ ↣ ∩ ∪ ⊆ ⊂ ⊄ ⊊ ⊇ ⊃ ⊅ ⊋ ⊖ ∈ ∉ ∋ ∌ ℕ ℤ ℚ ℝ ℂ ℵ ℶ ℷ ℸ 𝓟
¬ ∨ ∧ ⊕ → ← ⇒ ⇐ ⇔ ∀ ∃ ∄ ∴ ∵ ⊤ ⊥ ⊢ ⊨ ⫤ ⊣ … ⋯ ⋮ ⋰ ⋱
∫ ∬ ∭ ∮ ∯ ∰ ∇ ∆ δ ∂ ℱ ℒ ℓ
𝛢𝛼 𝛣𝛽 𝛤𝛾 𝛥𝛿 𝛦𝜀𝜖 𝛧𝜁 𝛨𝜂 𝛩𝜃𝜗 𝛪𝜄 𝛫𝜅 𝛬𝜆 𝛭𝜇 𝛮𝜈 𝛯𝜉 𝛰𝜊 𝛱𝜋 𝛲𝜌 𝛴𝜎𝜍 𝛵𝜏 𝛶𝜐 𝛷𝜙𝜑 𝛸𝜒 𝛹𝜓 𝛺𝜔