What is Stable Diffusion

<p style="color:red">注意：这篇博客中的图片存在 AI 生成的图片！</p>

Stable Diffusion is a deep learning text-to-image generation model released in 2022. It is primarily used to generate detailed images based on text descriptions, although it can also be applied to other tasks such as inpainting, outpainting, and image-to-image translation guided by prompts (English).

Why I Say "For Your Friends"

A Stable Diffusion service exposed to the public internet can get overwhelmed by massive computational demand, making it impossible for you and your friends to use it at all...

So a more ideal solution is to add a password, allowing only a few close friends to share your GPU resources. Let's enjoy using your drawing platform together!

file

Now your GPU purchase wasn't in vain!

Deciding Between Consumer-Grade and Enterprise-Grade Hardware

Consumer-grade hardware is cheap and simple, but it has limited load capacity and poor management capabilities.

Enterprise-grade hardware is slightly more complex, but it is easier to scale and manage.

Consumer-Grade Hardware?

Scaling out with consumer-grade hardware is difficult, especially since there is no BMC. The user experience is extremely poor. However, if you have a limited budget and light workloads, consumer-grade hardware can still handle the task.

Obviously, operating an AI painting system requires certain hardware.

file

You can refer to my hardware setup here:

  • ASUS TUF Gaming GeForce RTX™ 4090 24GB GDDR6X
  • ROG STRIX Z790-A GAMING WIFI
  • Intel® Core™ i9-13900KS Processor
  • SAMSUNG 990 PRO PCIe® 4.0 NVMe™ SSD 1TB

It runs smoothly without any issues.

Apply moderate overclocking. Refer to the tutorial: https://anduin.aiursoft.cn/post/2023/2/12/pc-tune

This tutorial is based on a consumer-grade hardware setup.

I strongly recommend purchasing a dedicated device specifically for running Stable Diffusion! Stable Diffusion requires a significant amount of VRAM! Even with my NVIDIA® GeForce RTX™ 4090, I still frequently run out of VRAM in daily use. If you have the budget, go for an A100.

Enterprise-Grade Hardware?

home server

For enterprise-grade hardware, you can opt to purchase a large number of cost-effective servers (such as the Dell R740) and then buy some compute cards, such as the P40 or A100.

Install ESXi on the servers to build a vSphere compute cluster. Be sure to configure vSAN.

Ensure that vMotion (dynamic migration of virtual machines) support is configured. Also, make sure to enable DRS (Distributed Resource Scheduler), HA (High Availability), and DPM (Distributed Power Management).

Purchase an Nvidia vGPU license. Use vCenter Lifecycle Manager to batch-deploy drivers and management daemons to the servers.

Create several Linux virtual machines on the servers and attach the vGPU to the compute cards. This will build a GPU farm.

Install the Nvidia drivers inside the virtual machines and activate the compute card licenses. Then, install Stable Diffusion.

Use a reverse proxy and load balancer on the outer layer to balance your computational load. Build a frontend app.

I have written a blog post on how to build a GPU compute farm: Add GPU Power to Your Data Center

Install Stable Diffusion

Prerequisites

You can use Ubuntu (recommended) or Windows Server. You can use a virtual machine and pass through either a vGPU or a physical GPU.

You can install google-perftools. This helps improve performance.

sudo apt-get install google-perftools

OpenGL may need to be installed (if you encounter a missing libgl error):

sudo apt-get install -y libgl1-mesa-glx

Python & Git required:

sudo apt install python3 python-is-python3 python3-pip git

Python 3.10.* must be installed.

file

Download Stable Diffusion WebUI

You can directly clone https://git.aiursoft.cn/PublicVault/stable-diffusion-webui

After the download is complete, simply run webui-user.bat and you are done.

It will run on 127.0.0.1:7860 by default. You can open it by double-clicking.

Download Some Checkpoints and LoRAs

There are several websites that host a large number of models, for example:

Civitai

file

Note: You may encounter pornographic and explicit content!!! Minors should not access this site.

After logging in, you can filter by Checkpoint and Lora.

When generating each image, you can only select one Checkpoint to use. However, you can combine and use multiple Loras simultaneously.

Using incompatible Lora combinations may result in bizarre and unpredictable outputs.

file

Place the Checkpoint and Lora files in the /models/Stable-diffusion and /models/Lora paths, respectively.

file

Try Your Hand with Ready-Made Data

Attempting to Recreate Classic Works from the Internet

You can find an existing image generated by someone else. Here, copy its generation data.

For example, I have randomly picked an image:

https://civitai.com/images/416564?modelVersionId=36644&prioritizedUserIds=75293&period=AllTime&sort=Most+Reactions&limit=20

On the website, the preview looks like this. Copy its generation data.

file

Paste the generation data into the Prompt. Then click: Read generation parameters

file

After this, all the parameters are already set.

Note, don't forget to manually select the correct Checkpoint! For example, the Checkpoint in the image is: 3Guofeng3.

file

If you haven't downloaded this Checkpoint in advance, the result will definitely be completely different.

Once all parameters are set and the Checkpoint is selected, just click generate, and you will see the exact same effect as the seller!

file

(I know that seeing these two images, you will definitely want to ask: Where is the butterfly? Where is the butterfly? Where is the butterfly? In reality, with these algorithms, Checkpoints, and Loras, even a small version difference, or a single missing letter in a prompt, can trigger a butterfly effect, resulting in a completely different image. This is normal.)

Trying to share your work parameters with others

If you have already generated a great-looking image and want to share your parameter settings with others so they can modify them further, you can simply copy and paste this section and send it to them.

file

After your friend receives the generated data above, they can use the method described above to quickly import the parameters and obtain an image identical to yours.

Attempting to retrieve generation data from the original image.

Sometimes, I generate a beautiful image of a girl, but I forget what parameters I used at the time.

file

If I want to review the parameters I used previously, I can use this trick.

Go directly to Png Info. Drag and drop the image into it.

file

Immediately, the list of parameters used to generate this image will appear on the right side.

file

You can click "Send to text2image" to edit the parameters used for generation.

Note: Don't forget to manually select the correct Checkpoint! For example, the Checkpoint in the image is: newmarsmix_R11.

Try Multiple Times

Try experimenting with different combinations of Checkpoints and Loras. By summarizing their patterns and stylistic characteristics, you will quickly become an expert.

file

Provide a Simpler, Safer UI

Directly exposing your local port to the internet is dangerous. Hackers can use features in Stable Diffusion to view the photos in your OneDrive. It is quite dangerous!

Disabling VENV

venv can create an isolated Python environment, separate from the system environment, which helps avoid conflicts between different versions of Python and its dependencies.

If your system is dedicated to running Stable Diffusion, disabling VENV can reduce deployment complexity.

You can directly use vim to modify webui.sh and remove the venv configuration function from that section.

Installing xformers

Xformers is a PyTorch-based library for implementing Transformer models. The Transformer is a popular neural network architecture commonly used in natural language processing tasks. Xformers is optional for Stable Diffusion.

Run directly in the directory where Stable Diffusion is located:

pip install xformers

Modifying Startup Parameters

Edit webui-user.bat to add some parameters. This makes the server better suited for exposure to the public internet.

@echo off

set PYTHON=
set GIT=
set VENV_DIR=
set COMMANDLINE_ARGS=--listen --hide-ui-dir-config --no-gradio-queue

call webui.bat

file

Of course, if you disable venv, running python launch.py directly works just fine too.

Disable Some Tabs

Edit config.json and change its hidden_tabs property to the following content. This prevents users from accessing specific features:

{
    "hidden_tabs": [
        "Settings",
        "Extensions",
        "Checkpoint Merger",
        "Train"
    ]
}

file

Setting Some Default and Configuration Values

Stable Diffusion allows batch generation by default.

However, exposing the batch generation feature to your friends is dangerous. If they casually generate a large batch, your electricity bill will skyrocket.

It is recommended to restrict these features.

Similarly, allowing your friends to enter arbitrary prompts is risky. If they casually enter nsfw content, you could end up in jail.

It is recommended to restrict Stable Diffusion's ability to generate explicit images.

Edit the ui-config.json file and modify the following properties:

{
    "txt2img/Prompt/visible": true,
    "txt2img/Prompt/value": "best quality, masterpiece",
    "txt2img/Negative prompt/visible": false,
    "txt2img/Negative prompt/value": "(((simple background))),nsfw,adult,porn,monochrome,lowres, bad anatomy, bad hands, text, error, missing fingers, extra digit, fewer digits, cropped, worst quality, low quality, normal quality, jpeg artifacts, signature, watermark, username, blurry, lowres, bad anatomy, bad hands, text, error, extra digit, fewer digits, cropped, worst quality, low quality, normal quality, jpeg artifacts, signature, watermark, username, blurry, ugly,pregnant,vore,duplicate,morbid,mut ilated,tran nsexual, hermaphrodite,long neck,mutated hands,poorly drawn hands,poorly drawn face,mutation,deformed,blurry,bad anatomy,bad proportions,malformed limbs,extra limbs,cloned face,disfigured,gross proportions, (((missing arms))),(((missing legs))), (((extra arms))),(((extra legs))),pubic hair, plump,bad legs,error legs,username,blurry,bad feet",
    "txt2img/Styles/visible": true,
    "txt2img/Styles/value": [],
    "txt2img/Sampling method/visible": true,
    "txt2img/Sampling method/value": "Euler a",// 可以选择你更适合的值。具体可以多种都试试。
    "txt2img/Sampling steps/visible": true,
    "txt2img/Sampling steps/value": 20,
    "txt2img/Sampling steps/minimum": 15,
    "txt2img/Sampling steps/maximum": 35,// 如果你算力冗余，可以拉大一点。
    "txt2img/Sampling steps/step": 1,
    "txt2img/Restore faces/visible": false,
    "txt2img/Restore faces/value": false,
    "txt2img/Tiling/visible": false,
    "txt2img/Tiling/value": false,
    "txt2img/Hires. fix/visible": true,
    "txt2img/Hires. fix/value": false, // 如果你算力冗余，可以改到 true
    "txt2img/Upscaler/visible": true,
    "txt2img/Upscaler/value": "R-ESRGAN 4x+ Anime6B",// 可以选择你更适合的值。具体可以多种都试试。
    "txt2img/Hires steps/visible": true,
    "txt2img/Hires steps/value": 32,
    "txt2img/Hires steps/minimum": 0,
    "txt2img/Hires steps/maximum": 128,// 如果你算力冗余，可以拉大一点。
    "txt2img/Hires steps/step": 1,
    "txt2img/Denoising strength/visible": true,
    "txt2img/Denoising strength/value": 0.5,
    "txt2img/Denoising strength/minimum": 0.0,
    "txt2img/Denoising strength/maximum": 1.0,
    "txt2img/Denoising strength/step": 0.01,
    "txt2img/Upscale by/visible": true,
    "txt2img/Upscale by/value": 2.0,
    "txt2img/Upscale by/minimum": 1.0,
    "txt2img/Upscale by/maximum": 4.0,// 如果你算力冗余，可以拉大一点。
    "txt2img/Upscale by/step": 0.05,
    "txt2img/Resize width to/visible": false,
    "txt2img/Resize width to/value": 0,
    "txt2img/Resize width to/minimum": 0,
    "txt2img/Resize width to/maximum": 2048,
    "txt2img/Resize width to/step": 8,
    "txt2img/Resize height to/visible": false,
    "txt2img/Resize height to/value": 0,
    "txt2img/Resize height to/minimum": 0,
    "txt2img/Resize height to/maximum": 2048,
    "txt2img/Resize height to/step": 8,
    "txt2img/Width/visible": true,
    "txt2img/Width/value": 640, // 如果你算力冗余，可以拉大一点。
    "txt2img/Width/minimum": 64,
    "txt2img/Width/maximum": 1024,// 如果你算力冗余，可以拉大一点。
    "txt2img/Width/step": 8,
    "txt2img/Height/visible": true,
    "txt2img/Height/value": 1024,// 如果你算力冗余，可以拉大一点。
    "txt2img/Height/minimum": 64,
    "txt2img/Height/maximum": 1024,// 如果你算力冗余，可以拉大一点。
    "txt2img/Height/step": 8,
    "txt2img/Batch count/visible": false,
    "txt2img/Batch count/value": 1,
    "txt2img/Batch count/minimum": 1,
    "txt2img/Batch count/maximum": 1,
    "txt2img/Batch count/step": 1,
    "txt2img/Batch size/visible": false,
    "txt2img/Batch size/value": 1,
    "txt2img/Batch size/minimum": 1,
    "txt2img/Batch size/maximum": 1,
    "txt2img/Batch size/step": 1,
    "txt2img/CFG Scale/visible": true,
    "txt2img/CFG Scale/value": 7.0,// 可以选择你更适合的值。具体可以多种都试试。
    "txt2img/CFG Scale/minimum": 3.0,
    "txt2img/CFG Scale/maximum": 15.0,
    "txt2img/CFG Scale/step": 1.0,
    "txt2img/Seed/visible": true,
    "txt2img/Seed/value": -1.0,
    "txt2img/Extra/visible": true,
    "txt2img/Extra/value": false,
    "txt2img/Variation seed/visible": true,
    "txt2img/Variation seed/value": -1.0,
    "txt2img/Variation strength/visible": true,
    "txt2img/Variation strength/value": 0.0,
    "txt2img/Variation strength/minimum": 0,
    "txt2img/Variation strength/maximum": 1,
    "txt2img/Variation strength/step": 0.01,
    "txt2img/Resize seed from width/visible": true,
    "txt2img/Resize seed from width/value": 0,
    "txt2img/Resize seed from width/minimum": 0,
    "txt2img/Resize seed from width/maximum": 2048,
    "txt2img/Resize seed from width/step": 8,
    "txt2img/Resize seed from height/visible": true,
    "txt2img/Resize seed from height/value": 0,
    "txt2img/Resize seed from height/minimum": 0,
    "txt2img/Resize seed from height/maximum": 2048,
    "txt2img/Resize seed from height/step": 8,
    "txt2img/Override settings/value": null,
    "txt2img/Script/visible": true,
    "txt2img/Script/value": "None"
}

Hide Some Buttons

Edit user.css and change its content to the following to hide some confusing buttons.


#txt2img_style_apply {
    display: none;
}

#txt2img_style_create {
    width: 0;
    height: 0;
    display: none;
}

#txt2img_clear_prompt {
    width: 0;
    height: 0;
    display: none;
}


#inpaint_tab {
    width: 0;
    height: 0;
    display: none;
}

#save_zip_txt2img {
    width: 0;
    height: 0;
    display: none;
}


#txt2img_subseed_show {
    width: 0;
    height: 0;
    display: none;
}

#txt2img_reuse_seed {
    width: 0;
    height: 0;
    display: none;
}

#txt2img_styles_row {
    width: 0;
    height: 0;
    display: none;
}

.progressDiv {
  display: block !important;
}

#txt2img_interrupt {
  width: 100% !important;
}

#txt2img_skip {
  display: none !important;
}

Finally, let's expose it to the public network!

Here, you can use the FRPC + Caddy approach to expose Stable Diffusion to the public internet with encrypted communication.

FRPC allows computers on the internet to actively connect to a local computer, even if it does not have a public IP address. Caddy is an excellent HTTP reverse proxy that can automatically obtain TLS certificates for web services.

For example, the result of my setup is: https://stable-diffusion.aiursoft.cn/

You can read the FRP tutorial here: FRP

You can read the Caddy tutorial here: Caddy

Of course, don’t forget to set up a password by following the Caddy tutorial.