Started by timer
Running as SYSTEM
Building in workspace /var/lib/jenkins/jobs/tensorflow/workspace
[SSH] script:
module load anaconda3_gpu
module load cudnn
cd python
rm -f tensorflow_test.txt
srun --nodes=1 --ntasks-per-node=1 --mem=16g \
--time=00:05:00 \
--partition=a100rome \
--account=bbmb-hydro \
--gpus-per-node=1 \
--gpu-bind=closest \
python3 cifar10gpu.py | tee tensorflow_test.txt
YVALUE=`grep "wall time seconds" tensorflow_test.txt | cut -d'=' -f2`
echo YVALUE=$YVALUE > tensorflow_time.txt
[SSH] executing...
srun: job 98654 queued and waiting for resources
srun: job 98654 has been allocated resources
2024-11-26 07:43:02.981382: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-11-26 07:43:04.247736: E tensorflow/stream_executor/cuda/cuda_blas.cc:2981] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2024-11-26 07:43:07.234614: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /sw/spack/hydrogpu-2022-06/apps/cudnn/8.4.0.27-11.6-gcc-11.3.0-qbhsfav/lib:/sw/spack/hydrogpu-2022-06/apps/cuda/11.7.0-gcc-11.3.0-3ysno6b/lib64:/sw/spack/hydrogpu-2022-06/apps/openmpi/4.1.4-gcc-11.3.0-i6koipj/lib:/sw/spack/hydrogpu-2022-06/apps/gcc/11.3.0-gcc-8.4.1-kroofxa/lib64:/sw/spack/hydrogpu-2022-06/apps/gcc/11.3.0-gcc-8.4.1-kroofxa/lib
2024-11-26 07:43:07.236039: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /sw/spack/hydrogpu-2022-06/apps/cudnn/8.4.0.27-11.6-gcc-11.3.0-qbhsfav/lib:/sw/spack/hydrogpu-2022-06/apps/cuda/11.7.0-gcc-11.3.0-3ysno6b/lib64:/sw/spack/hydrogpu-2022-06/apps/openmpi/4.1.4-gcc-11.3.0-i6koipj/lib:/sw/spack/hydrogpu-2022-06/apps/gcc/11.3.0-gcc-8.4.1-kroofxa/lib64:/sw/spack/hydrogpu-2022-06/apps/gcc/11.3.0-gcc-8.4.1-kroofxa/lib
2024-11-26 07:43:07.236048: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
2024-11-26 07:43:14.130811: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-11-26 07:43:15.411487: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1616] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 79148 MB memory: -> device: 0, name: NVIDIA A100 80GB PCIe, pci bus id: 0000:21:00.0, compute capability: 8.0
(50000, 32, 32, 3) (10000, 32, 32, 3)
Epoch 1/10
2024-11-26 07:43:17.748833: I tensorflow/stream_executor/cuda/cuda_blas.cc:1614] TensorFloat-32 will be used for the matrix multiplication. This will only be logged once.
1/1563 [..............................] - ETA: 29:56 - loss: 2.3440 - accuracy: 0.0625
37/1563 [..............................] - ETA: 2s - loss: 2.2482 - accuracy: 0.1588
71/1563 [>.............................] - ETA: 2s - loss: 2.1773 - accuracy: 0.1945
108/1563 [=>............................] - ETA: 2s - loss: 2.1238 - accuracy: 0.2167
145/1563 [=>............................] - ETA: 1s - loss: 2.0838 - accuracy: 0.2362
182/1563 [==>...........................] - ETA: 1s - loss: 2.0573 - accuracy: 0.2485
219/1563 [===>..........................] - ETA: 1s - loss: 2.0388 - accuracy: 0.2574
256/1563 [===>..........................] - ETA: 1s - loss: 2.0187 - accuracy: 0.2676
293/1563 [====>.........................] - ETA: 1s - loss: 2.0068 - accuracy: 0.2720
330/1563 [=====>........................] - ETA: 1s - loss: 1.9925 - accuracy: 0.2779
368/1563 [======>.......................] - ETA: 1s - loss: 1.9755 - accuracy: 0.2852
406/1563 [======>.......................] - ETA: 1s - loss: 1.9638 - accuracy: 0.2907
444/1563 [=======>......................] - ETA: 1s - loss: 1.9542 - accuracy: 0.2956
482/1563 [========>.....................] - ETA: 1s - loss: 1.9447 - accuracy: 0.2975
520/1563 [========>.....................] - ETA: 1s - loss: 1.9391 - accuracy: 0.3002
558/1563 [=========>....................] - ETA: 1s - loss: 1.9258 - accuracy: 0.3061
596/1563 [==========>...................] - ETA: 1s - loss: 1.9166 - accuracy: 0.3086
634/1563 [===========>..................] - ETA: 1s - loss: 1.9110 - accuracy: 0.3117
672/1563 [===========>..................] - ETA: 1s - loss: 1.9042 - accuracy: 0.3141
709/1563 [============>.................] - ETA: 1s - loss: 1.8981 - accuracy: 0.3163
746/1563 [=============>................] - ETA: 1s - loss: 1.8946 - accuracy: 0.3184
784/1563 [==============>...............] - ETA: 1s - loss: 1.8895 - accuracy: 0.3205
822/1563 [==============>...............] - ETA: 1s - loss: 1.8839 - accuracy: 0.3221
860/1563 [===============>..............] - ETA: 0s - loss: 1.8782 - accuracy: 0.3239
898/1563 [================>.............] - ETA: 0s - loss: 1.8732 - accuracy: 0.3259
935/1563 [================>.............] - ETA: 0s - loss: 1.8695 - accuracy: 0.3287
973/1563 [=================>............] - ETA: 0s - loss: 1.8637 - accuracy: 0.3310
1011/1563 [==================>...........] - ETA: 0s - loss: 1.8607 - accuracy: 0.3325
1049/1563 [===================>..........] - ETA: 0s - loss: 1.8562 - accuracy: 0.3346
1087/1563 [===================>..........] - ETA: 0s - loss: 1.8512 - accuracy: 0.3369
1122/1563 [====================>.........] - ETA: 0s - loss: 1.8474 - accuracy: 0.3387
1160/1563 [=====================>........] - ETA: 0s - loss: 1.8443 - accuracy: 0.3400
1198/1563 [=====================>........] - ETA: 0s - loss: 1.8412 - accuracy: 0.3408
1236/1563 [======================>.......] - ETA: 0s - loss: 1.8370 - accuracy: 0.3423
1274/1563 [=======================>......] - ETA: 0s - loss: 1.8316 - accuracy: 0.3443
1312/1563 [========================>.....] - ETA: 0s - loss: 1.8283 - accuracy: 0.3458
1350/1563 [========================>.....] - ETA: 0s - loss: 1.8258 - accuracy: 0.3468
1388/1563 [=========================>....] - ETA: 0s - loss: 1.8219 - accuracy: 0.3481
1426/1563 [==========================>...] - ETA: 0s - loss: 1.8186 - accuracy: 0.3493
1464/1563 [===========================>..] - ETA: 0s - loss: 1.8154 - accuracy: 0.3505
1502/1563 [===========================>..] - ETA: 0s - loss: 1.8125 - accuracy: 0.3518
1539/1563 [============================>.] - ETA: 0s - loss: 1.8090 - accuracy: 0.3529
1563/1563 [==============================] - 3s 1ms/step - loss: 1.8071 - accuracy: 0.3535
Epoch 2/10
1/1563 [..............................] - ETA: 2s - loss: 1.7153 - accuracy: 0.3750
38/1563 [..............................] - ETA: 2s - loss: 1.6635 - accuracy: 0.4095
75/1563 [>.............................] - ETA: 2s - loss: 1.6842 - accuracy: 0.4050
112/1563 [=>............................] - ETA: 1s - loss: 1.6867 - accuracy: 0.4015
149/1563 [=>............................] - ETA: 1s - loss: 1.6833 - accuracy: 0.3995
186/1563 [==>...........................] - ETA: 1s - loss: 1.6755 - accuracy: 0.4041
223/1563 [===>..........................] - ETA: 1s - loss: 1.6691 - accuracy: 0.4072
261/1563 [====>.........................] - ETA: 1s - loss: 1.6644 - accuracy: 0.4085
299/1563 [====>.........................] - ETA: 1s - loss: 1.6608 - accuracy: 0.4105
337/1563 [=====>........................] - ETA: 1s - loss: 1.6621 - accuracy: 0.4084
375/1563 [======>.......................] - ETA: 1s - loss: 1.6589 - accuracy: 0.4108
413/1563 [======>.......................] - ETA: 1s - loss: 1.6511 - accuracy: 0.4119
451/1563 [=======>......................] - ETA: 1s - loss: 1.6522 - accuracy: 0.4122
489/1563 [========>.....................] - ETA: 1s - loss: 1.6489 - accuracy: 0.4140
527/1563 [=========>....................] - ETA: 1s - loss: 1.6432 - accuracy: 0.4167
565/1563 [=========>....................] - ETA: 1s - loss: 1.6429 - accuracy: 0.4159
603/1563 [==========>...................] - ETA: 1s - loss: 1.6400 - accuracy: 0.4171
641/1563 [===========>..................] - ETA: 1s - loss: 1.6381 - accuracy: 0.4180
679/1563 [============>.................] - ETA: 1s - loss: 1.6358 - accuracy: 0.4182
717/1563 [============>.................] - ETA: 1s - loss: 1.6378 - accuracy: 0.4175
754/1563 [=============>................] - ETA: 1s - loss: 1.6387 - accuracy: 0.4180
792/1563 [==============>...............] - ETA: 1s - loss: 1.6367 - accuracy: 0.4181
831/1563 [==============>...............] - ETA: 0s - loss: 1.6343 - accuracy: 0.4201
869/1563 [===============>..............] - ETA: 0s - loss: 1.6332 - accuracy: 0.4210
906/1563 [================>.............] - ETA: 0s - loss: 1.6332 - accuracy: 0.4208
943/1563 [=================>............] - ETA: 0s - loss: 1.6328 - accuracy: 0.4206
980/1563 [=================>............] - ETA: 0s - loss: 1.6315 - accuracy: 0.4207
1017/1563 [==================>...........] - ETA: 0s - loss: 1.6307 - accuracy: 0.4216
1054/1563 [===================>..........] - ETA: 0s - loss: 1.6309 - accuracy: 0.4224
1091/1563 [===================>..........] - ETA: 0s - loss: 1.6292 - accuracy: 0.4234
1128/1563 [====================>.........] - ETA: 0s - loss: 1.6309 - accuracy: 0.4227
1165/1563 [=====================>........] - ETA: 0s - loss: 1.6300 - accuracy: 0.4234
1202/1563 [======================>.......] - ETA: 0s - loss: 1.6279 - accuracy: 0.4243
1239/1563 [======================>.......] - ETA: 0s - loss: 1.6266 - accuracy: 0.4249
1276/1563 [=======================>......] - ETA: 0s - loss: 1.6246 - accuracy: 0.4251
1313/1563 [========================>.....] - ETA: 0s - loss: 1.6226 - accuracy: 0.4257
1350/1563 [========================>.....] - ETA: 0s - loss: 1.6227 - accuracy: 0.4257
1387/1563 [=========================>....] - ETA: 0s - loss: 1.6221 - accuracy: 0.4263
1424/1563 [==========================>...] - ETA: 0s - loss: 1.6211 - accuracy: 0.4270
1461/1563 [===========================>..] - ETA: 0s - loss: 1.6205 - accuracy: 0.4273
1498/1563 [===========================>..] - ETA: 0s - loss: 1.6208 - accuracy: 0.4271
1534/1563 [============================>.] - ETA: 0s - loss: 1.6198 - accuracy: 0.4270
1563/1563 [==============================] - 2s 1ms/step - loss: 1.6196 - accuracy: 0.4270
Epoch 3/10
1/1563 [..............................] - ETA: 2s - loss: 1.5870 - accuracy: 0.4062
38/1563 [..............................] - ETA: 2s - loss: 1.5448 - accuracy: 0.4572
75/1563 [>.............................] - ETA: 2s - loss: 1.5625 - accuracy: 0.4546
112/1563 [=>............................] - ETA: 1s - loss: 1.5495 - accuracy: 0.4537
149/1563 [=>............................] - ETA: 1s - loss: 1.5476 - accuracy: 0.4526
186/1563 [==>...........................] - ETA: 1s - loss: 1.5541 - accuracy: 0.4496
224/1563 [===>..........................] - ETA: 1s - loss: 1.5488 - accuracy: 0.4515
262/1563 [====>.........................] - ETA: 1s - loss: 1.5461 - accuracy: 0.4526
300/1563 [====>.........................] - ETA: 1s - loss: 1.5493 - accuracy: 0.4531
338/1563 [=====>........................] - ETA: 1s - loss: 1.5490 - accuracy: 0.4542
376/1563 [======>.......................] - ETA: 1s - loss: 1.5511 - accuracy: 0.4544
413/1563 [======>.......................] - ETA: 1s - loss: 1.5498 - accuracy: 0.4564
450/1563 [=======>......................] - ETA: 1s - loss: 1.5479 - accuracy: 0.4569
488/1563 [========>.....................] - ETA: 1s - loss: 1.5491 - accuracy: 0.4545
526/1563 [=========>....................] - ETA: 1s - loss: 1.5467 - accuracy: 0.4556
564/1563 [=========>....................] - ETA: 1s - loss: 1.5440 - accuracy: 0.4570
602/1563 [==========>...................] - ETA: 1s - loss: 1.5451 - accuracy: 0.4569
639/1563 [===========>..................] - ETA: 1s - loss: 1.5441 - accuracy: 0.4573
677/1563 [===========>..................] - ETA: 1s - loss: 1.5446 - accuracy: 0.4565
715/1563 [============>.................] - ETA: 1s - loss: 1.5442 - accuracy: 0.4566
752/1563 [=============>................] - ETA: 1s - loss: 1.5423 - accuracy: 0.4567
789/1563 [==============>...............] - ETA: 1s - loss: 1.5424 - accuracy: 0.4560
826/1563 [==============>...............] - ETA: 0s - loss: 1.5428 - accuracy: 0.4558
863/1563 [===============>..............] - ETA: 0s - loss: 1.5409 - accuracy: 0.4559
901/1563 [================>.............] - ETA: 0s - loss: 1.5429 - accuracy: 0.4549
938/1563 [=================>............] - ETA: 0s - loss: 1.5427 - accuracy: 0.4545
975/1563 [=================>............] - ETA: 0s - loss: 1.5417 - accuracy: 0.4547
1012/1563 [==================>...........] - ETA: 0s - loss: 1.5437 - accuracy: 0.4541
1049/1563 [===================>..........] - ETA: 0s - loss: 1.5449 - accuracy: 0.4542
1086/1563 [===================>..........] - ETA: 0s - loss: 1.5430 - accuracy: 0.4548
1123/1563 [====================>.........] - ETA: 0s - loss: 1.5421 - accuracy: 0.4556
1160/1563 [=====================>........] - ETA: 0s - loss: 1.5433 - accuracy: 0.4553
1198/1563 [=====================>........] - ETA: 0s - loss: 1.5435 - accuracy: 0.4556
1236/1563 [======================>.......] - ETA: 0s - loss: 1.5431 - accuracy: 0.4560
1273/1563 [=======================>......] - ETA: 0s - loss: 1.5420 - accuracy: 0.4558
1311/1563 [========================>.....] - ETA: 0s - loss: 1.5410 - accuracy: 0.4569
1349/1563 [========================>.....] - ETA: 0s - loss: 1.5407 - accuracy: 0.4567
1386/1563 [=========================>....] - ETA: 0s - loss: 1.5410 - accuracy: 0.4570
1423/1563 [==========================>...] - ETA: 0s - loss: 1.5408 - accuracy: 0.4566
1461/1563 [===========================>..] - ETA: 0s - loss: 1.5388 - accuracy: 0.4576
1499/1563 [===========================>..] - ETA: 0s - loss: 1.5392 - accuracy: 0.4578
1535/1563 [============================>.] - ETA: 0s - loss: 1.5395 - accuracy: 0.4576
1563/1563 [==============================] - 2s 1ms/step - loss: 1.5388 - accuracy: 0.4581
Epoch 4/10
1/1563 [..............................] - ETA: 2s - loss: 1.5013 - accuracy: 0.4375
38/1563 [..............................] - ETA: 2s - loss: 1.4835 - accuracy: 0.4836
76/1563 [>.............................] - ETA: 2s - loss: 1.5113 - accuracy: 0.4725
113/1563 [=>............................] - ETA: 1s - loss: 1.5087 - accuracy: 0.4710
151/1563 [=>............................] - ETA: 1s - loss: 1.5133 - accuracy: 0.4708
189/1563 [==>...........................] - ETA: 1s - loss: 1.5008 - accuracy: 0.4770
226/1563 [===>..........................] - ETA: 1s - loss: 1.4986 - accuracy: 0.4755
263/1563 [====>.........................] - ETA: 1s - loss: 1.4981 - accuracy: 0.4723
300/1563 [====>.........................] - ETA: 1s - loss: 1.4946 - accuracy: 0.4723
337/1563 [=====>........................] - ETA: 1s - loss: 1.4925 - accuracy: 0.4726
374/1563 [======>.......................] - ETA: 1s - loss: 1.4908 - accuracy: 0.4739
411/1563 [======>.......................] - ETA: 1s - loss: 1.4925 - accuracy: 0.4710
449/1563 [=======>......................] - ETA: 1s - loss: 1.4933 - accuracy: 0.4715
487/1563 [========>.....................] - ETA: 1s - loss: 1.4921 - accuracy: 0.4720
525/1563 [=========>....................] - ETA: 1s - loss: 1.4907 - accuracy: 0.4724
562/1563 [=========>....................] - ETA: 1s - loss: 1.4910 - accuracy: 0.4735
599/1563 [==========>...................] - ETA: 1s - loss: 1.4942 - accuracy: 0.4728
636/1563 [===========>..................] - ETA: 1s - loss: 1.4966 - accuracy: 0.4716
673/1563 [===========>..................] - ETA: 1s - loss: 1.4960 - accuracy: 0.4710
710/1563 [============>.................] - ETA: 1s - loss: 1.4967 - accuracy: 0.4717
748/1563 [=============>................] - ETA: 1s - loss: 1.4934 - accuracy: 0.4725
786/1563 [==============>...............] - ETA: 1s - loss: 1.4925 - accuracy: 0.4729
824/1563 [==============>...............] - ETA: 0s - loss: 1.4941 - accuracy: 0.4727
862/1563 [===============>..............] - ETA: 0s - loss: 1.4924 - accuracy: 0.4731
900/1563 [================>.............] - ETA: 0s - loss: 1.4925 - accuracy: 0.4728
938/1563 [=================>............] - ETA: 0s - loss: 1.4922 - accuracy: 0.4732
975/1563 [=================>............] - ETA: 0s - loss: 1.4903 - accuracy: 0.4743
1011/1563 [==================>...........] - ETA: 0s - loss: 1.4882 - accuracy: 0.4753
1049/1563 [===================>..........] - ETA: 0s - loss: 1.4890 - accuracy: 0.4751
1086/1563 [===================>..........] - ETA: 0s - loss: 1.4895 - accuracy: 0.4740
1124/1563 [====================>.........] - ETA: 0s - loss: 1.4884 - accuracy: 0.4745
1161/1563 [=====================>........] - ETA: 0s - loss: 1.4863 - accuracy: 0.4758
1198/1563 [=====================>........] - ETA: 0s - loss: 1.4857 - accuracy: 0.4762
1235/1563 [======================>.......] - ETA: 0s - loss: 1.4840 - accuracy: 0.4771
1272/1563 [=======================>......] - ETA: 0s - loss: 1.4827 - accuracy: 0.4778
1309/1563 [========================>.....] - ETA: 0s - loss: 1.4827 - accuracy: 0.4774
1347/1563 [========================>.....] - ETA: 0s - loss: 1.4822 - accuracy: 0.4773
1385/1563 [=========================>....] - ETA: 0s - loss: 1.4828 - accuracy: 0.4772
1423/1563 [==========================>...] - ETA: 0s - loss: 1.4824 - accuracy: 0.4769
1460/1563 [===========================>..] - ETA: 0s - loss: 1.4817 - accuracy: 0.4773
1497/1563 [===========================>..] - ETA: 0s - loss: 1.4801 - accuracy: 0.4774
1534/1563 [============================>.] - ETA: 0s - loss: 1.4801 - accuracy: 0.4775
1563/1563 [==============================] - 2s 1ms/step - loss: 1.4804 - accuracy: 0.4775
Epoch 5/10
1/1563 [..............................] - ETA: 2s - loss: 1.7573 - accuracy: 0.3750
39/1563 [..............................] - ETA: 2s - loss: 1.3996 - accuracy: 0.5032
76/1563 [>.............................] - ETA: 2s - loss: 1.4049 - accuracy: 0.5049
114/1563 [=>............................] - ETA: 1s - loss: 1.4404 - accuracy: 0.4929
151/1563 [=>............................] - ETA: 1s - loss: 1.4366 - accuracy: 0.4925
187/1563 [==>...........................] - ETA: 1s - loss: 1.4413 - accuracy: 0.4886
225/1563 [===>..........................] - ETA: 1s - loss: 1.4428 - accuracy: 0.4860
263/1563 [====>.........................] - ETA: 1s - loss: 1.4396 - accuracy: 0.4879
300/1563 [====>.........................] - ETA: 1s - loss: 1.4385 - accuracy: 0.4885
337/1563 [=====>........................] - ETA: 1s - loss: 1.4449 - accuracy: 0.4879
375/1563 [======>.......................] - ETA: 1s - loss: 1.4477 - accuracy: 0.4877
413/1563 [======>.......................] - ETA: 1s - loss: 1.4397 - accuracy: 0.4917
451/1563 [=======>......................] - ETA: 1s - loss: 1.4390 - accuracy: 0.4914
486/1563 [========>.....................] - ETA: 1s - loss: 1.4405 - accuracy: 0.4916
523/1563 [=========>....................] - ETA: 1s - loss: 1.4414 - accuracy: 0.4913
561/1563 [=========>....................] - ETA: 1s - loss: 1.4403 - accuracy: 0.4919
599/1563 [==========>...................] - ETA: 1s - loss: 1.4422 - accuracy: 0.4911
636/1563 [===========>..................] - ETA: 1s - loss: 1.4410 - accuracy: 0.4917
673/1563 [===========>..................] - ETA: 1s - loss: 1.4394 - accuracy: 0.4918
711/1563 [============>.................] - ETA: 1s - loss: 1.4396 - accuracy: 0.4923
748/1563 [=============>................] - ETA: 1s - loss: 1.4391 - accuracy: 0.4927
786/1563 [==============>...............] - ETA: 1s - loss: 1.4367 - accuracy: 0.4938
823/1563 [==============>...............] - ETA: 1s - loss: 1.4349 - accuracy: 0.4951
860/1563 [===============>..............] - ETA: 0s - loss: 1.4354 - accuracy: 0.4957
898/1563 [================>.............] - ETA: 0s - loss: 1.4348 - accuracy: 0.4959
935/1563 [================>.............] - ETA: 0s - loss: 1.4353 - accuracy: 0.4955
972/1563 [=================>............] - ETA: 0s - loss: 1.4359 - accuracy: 0.4960
1009/1563 [==================>...........] - ETA: 0s - loss: 1.4348 - accuracy: 0.4962
1047/1563 [===================>..........] - ETA: 0s - loss: 1.4331 - accuracy: 0.4975
1084/1563 [===================>..........] - ETA: 0s - loss: 1.4326 - accuracy: 0.4975
1122/1563 [====================>.........] - ETA: 0s - loss: 1.4324 - accuracy: 0.4979
1159/1563 [=====================>........] - ETA: 0s - loss: 1.4317 - accuracy: 0.4976
1196/1563 [=====================>........] - ETA: 0s - loss: 1.4308 - accuracy: 0.4979
1233/1563 [======================>.......] - ETA: 0s - loss: 1.4306 - accuracy: 0.4982
1270/1563 [=======================>......] - ETA: 0s - loss: 1.4315 - accuracy: 0.4981
1307/1563 [========================>.....] - ETA: 0s - loss: 1.4314 - accuracy: 0.4978
1344/1563 [========================>.....] - ETA: 0s - loss: 1.4310 - accuracy: 0.4979
1382/1563 [=========================>....] - ETA: 0s - loss: 1.4307 - accuracy: 0.4979
1420/1563 [==========================>...] - ETA: 0s - loss: 1.4311 - accuracy: 0.4978
1458/1563 [==========================>...] - ETA: 0s - loss: 1.4317 - accuracy: 0.4972
1495/1563 [===========================>..] - ETA: 0s - loss: 1.4314 - accuracy: 0.4976
1532/1563 [============================>.] - ETA: 0s - loss: 1.4307 - accuracy: 0.4977
1563/1563 [==============================] - 2s 1ms/step - loss: 1.4307 - accuracy: 0.4979
Epoch 6/10
1/1563 [..............................] - ETA: 2s - loss: 1.6081 - accuracy: 0.3750
38/1563 [..............................] - ETA: 2s - loss: 1.4195 - accuracy: 0.4852
75/1563 [>.............................] - ETA: 2s - loss: 1.4222 - accuracy: 0.4892
112/1563 [=>............................] - ETA: 1s - loss: 1.4130 - accuracy: 0.4953
147/1563 [=>............................] - ETA: 1s - loss: 1.4064 - accuracy: 0.5004
183/1563 [==>...........................] - ETA: 1s - loss: 1.3908 - accuracy: 0.5073
220/1563 [===>..........................] - ETA: 1s - loss: 1.3904 - accuracy: 0.5080
258/1563 [===>..........................] - ETA: 1s - loss: 1.3860 - accuracy: 0.5097
296/1563 [====>.........................] - ETA: 1s - loss: 1.3899 - accuracy: 0.5092
333/1563 [=====>........................] - ETA: 1s - loss: 1.3936 - accuracy: 0.5082
370/1563 [======>.......................] - ETA: 1s - loss: 1.3933 - accuracy: 0.5076
408/1563 [======>.......................] - ETA: 1s - loss: 1.3951 - accuracy: 0.5083
446/1563 [=======>......................] - ETA: 1s - loss: 1.3883 - accuracy: 0.5109
484/1563 [========>.....................] - ETA: 1s - loss: 1.3905 - accuracy: 0.5119
522/1563 [=========>....................] - ETA: 1s - loss: 1.3938 - accuracy: 0.5098
560/1563 [=========>....................] - ETA: 1s - loss: 1.3913 - accuracy: 0.5103
598/1563 [==========>...................] - ETA: 1s - loss: 1.3895 - accuracy: 0.5106
635/1563 [===========>..................] - ETA: 1s - loss: 1.3901 - accuracy: 0.5109
672/1563 [===========>..................] - ETA: 1s - loss: 1.3911 - accuracy: 0.5109
709/1563 [============>.................] - ETA: 1s - loss: 1.3900 - accuracy: 0.5112
746/1563 [=============>................] - ETA: 1s - loss: 1.3890 - accuracy: 0.5108
783/1563 [==============>...............] - ETA: 1s - loss: 1.3925 - accuracy: 0.5093
821/1563 [==============>...............] - ETA: 1s - loss: 1.3902 - accuracy: 0.5100
859/1563 [===============>..............] - ETA: 0s - loss: 1.3901 - accuracy: 0.5098
896/1563 [================>.............] - ETA: 0s - loss: 1.3904 - accuracy: 0.5095
933/1563 [================>.............] - ETA: 0s - loss: 1.3898 - accuracy: 0.5095
970/1563 [=================>............] - ETA: 0s - loss: 1.3903 - accuracy: 0.5101
1008/1563 [==================>...........] - ETA: 0s - loss: 1.3896 - accuracy: 0.5108
1045/1563 [===================>..........] - ETA: 0s - loss: 1.3908 - accuracy: 0.5104
1083/1563 [===================>..........] - ETA: 0s - loss: 1.3902 - accuracy: 0.5108
1121/1563 [====================>.........] - ETA: 0s - loss: 1.3907 - accuracy: 0.5101
1159/1563 [=====================>........] - ETA: 0s - loss: 1.3905 - accuracy: 0.5105
1197/1563 [=====================>........] - ETA: 0s - loss: 1.3904 - accuracy: 0.5108
1235/1563 [======================>.......] - ETA: 0s - loss: 1.3893 - accuracy: 0.5114
1272/1563 [=======================>......] - ETA: 0s - loss: 1.3883 - accuracy: 0.5117
1309/1563 [========================>.....] - ETA: 0s - loss: 1.3884 - accuracy: 0.5118
1346/1563 [========================>.....] - ETA: 0s - loss: 1.3873 - accuracy: 0.5124
1383/1563 [=========================>....] - ETA: 0s - loss: 1.3867 - accuracy: 0.5126
1421/1563 [==========================>...] - ETA: 0s - loss: 1.3873 - accuracy: 0.5126
1458/1563 [==========================>...] - ETA: 0s - loss: 1.3868 - accuracy: 0.5128
1495/1563 [===========================>..] - ETA: 0s - loss: 1.3865 - accuracy: 0.5127
1532/1563 [============================>.] - ETA: 0s - loss: 1.3861 - accuracy: 0.5129
1563/1563 [==============================] - 2s 1ms/step - loss: 1.3861 - accuracy: 0.5128
Epoch 7/10
1/1563 [..............................] - ETA: 2s - loss: 1.4714 - accuracy: 0.5938
38/1563 [..............................] - ETA: 2s - loss: 1.3489 - accuracy: 0.5238
76/1563 [>.............................] - ETA: 1s - loss: 1.3583 - accuracy: 0.5214
114/1563 [=>............................] - ETA: 1s - loss: 1.3627 - accuracy: 0.5206
152/1563 [=>............................] - ETA: 1s - loss: 1.3595 - accuracy: 0.5212
190/1563 [==>...........................] - ETA: 1s - loss: 1.3583 - accuracy: 0.5235
227/1563 [===>..........................] - ETA: 1s - loss: 1.3595 - accuracy: 0.5230
264/1563 [====>.........................] - ETA: 1s - loss: 1.3531 - accuracy: 0.5244
303/1563 [====>.........................] - ETA: 1s - loss: 1.3512 - accuracy: 0.5226
341/1563 [=====>........................] - ETA: 1s - loss: 1.3528 - accuracy: 0.5228
379/1563 [======>.......................] - ETA: 1s - loss: 1.3528 - accuracy: 0.5202
417/1563 [=======>......................] - ETA: 1s - loss: 1.3503 - accuracy: 0.5192
456/1563 [=======>......................] - ETA: 1s - loss: 1.3500 - accuracy: 0.5197
495/1563 [========>.....................] - ETA: 1s - loss: 1.3548 - accuracy: 0.5185
533/1563 [=========>....................] - ETA: 1s - loss: 1.3523 - accuracy: 0.5200
571/1563 [=========>....................] - ETA: 1s - loss: 1.3498 - accuracy: 0.5217
610/1563 [==========>...................] - ETA: 1s - loss: 1.3505 - accuracy: 0.5223
647/1563 [===========>..................] - ETA: 1s - loss: 1.3536 - accuracy: 0.5219
684/1563 [============>.................] - ETA: 1s - loss: 1.3549 - accuracy: 0.5211
722/1563 [============>.................] - ETA: 1s - loss: 1.3544 - accuracy: 0.5220
760/1563 [=============>................] - ETA: 1s - loss: 1.3551 - accuracy: 0.5224
798/1563 [==============>...............] - ETA: 1s - loss: 1.3550 - accuracy: 0.5227
836/1563 [===============>..............] - ETA: 0s - loss: 1.3546 - accuracy: 0.5223
874/1563 [===============>..............] - ETA: 0s - loss: 1.3537 - accuracy: 0.5226
911/1563 [================>.............] - ETA: 0s - loss: 1.3536 - accuracy: 0.5225
948/1563 [=================>............] - ETA: 0s - loss: 1.3525 - accuracy: 0.5228
985/1563 [=================>............] - ETA: 0s - loss: 1.3527 - accuracy: 0.5230
1023/1563 [==================>...........] - ETA: 0s - loss: 1.3522 - accuracy: 0.5229
1061/1563 [===================>..........] - ETA: 0s - loss: 1.3523 - accuracy: 0.5228
1099/1563 [====================>.........] - ETA: 0s - loss: 1.3533 - accuracy: 0.5226
1138/1563 [====================>.........] - ETA: 0s - loss: 1.3535 - accuracy: 0.5230
1176/1563 [=====================>........] - ETA: 0s - loss: 1.3538 - accuracy: 0.5229
1213/1563 [======================>.......] - ETA: 0s - loss: 1.3526 - accuracy: 0.5233
1250/1563 [======================>.......] - ETA: 0s - loss: 1.3526 - accuracy: 0.5229
1287/1563 [=======================>......] - ETA: 0s - loss: 1.3517 - accuracy: 0.5235
1325/1563 [========================>.....] - ETA: 0s - loss: 1.3519 - accuracy: 0.5237
1363/1563 [=========================>....] - ETA: 0s - loss: 1.3508 - accuracy: 0.5238
1402/1563 [=========================>....] - ETA: 0s - loss: 1.3514 - accuracy: 0.5236
1441/1563 [==========================>...] - ETA: 0s - loss: 1.3512 - accuracy: 0.5233
1479/1563 [===========================>..] - ETA: 0s - loss: 1.3507 - accuracy: 0.5235
1518/1563 [============================>.] - ETA: 0s - loss: 1.3517 - accuracy: 0.5236
1556/1563 [============================>.] - ETA: 0s - loss: 1.3516 - accuracy: 0.5236
1563/1563 [==============================] - 2s 1ms/step - loss: 1.3517 - accuracy: 0.5235
Epoch 8/10
1/1563 [..............................] - ETA: 2s - loss: 1.0900 - accuracy: 0.6250
39/1563 [..............................] - ETA: 2s - loss: 1.3618 - accuracy: 0.5321
77/1563 [>.............................] - ETA: 2s - loss: 1.3490 - accuracy: 0.5296
115/1563 [=>............................] - ETA: 1s - loss: 1.3350 - accuracy: 0.5321
153/1563 [=>............................] - ETA: 1s - loss: 1.3387 - accuracy: 0.5302
191/1563 [==>...........................] - ETA: 1s - loss: 1.3261 - accuracy: 0.5334
228/1563 [===>..........................] - ETA: 1s - loss: 1.3106 - accuracy: 0.5381
265/1563 [====>.........................] - ETA: 1s - loss: 1.3158 - accuracy: 0.5357
302/1563 [====>.........................] - ETA: 1s - loss: 1.3213 - accuracy: 0.5338
339/1563 [=====>........................] - ETA: 1s - loss: 1.3216 - accuracy: 0.5347
377/1563 [======>.......................] - ETA: 1s - loss: 1.3214 - accuracy: 0.5357
415/1563 [======>.......................] - ETA: 1s - loss: 1.3142 - accuracy: 0.5373
453/1563 [=======>......................] - ETA: 1s - loss: 1.3139 - accuracy: 0.5366
490/1563 [========>.....................] - ETA: 1s - loss: 1.3174 - accuracy: 0.5354
527/1563 [=========>....................] - ETA: 1s - loss: 1.3173 - accuracy: 0.5358
565/1563 [=========>....................] - ETA: 1s - loss: 1.3154 - accuracy: 0.5352
602/1563 [==========>...................] - ETA: 1s - loss: 1.3150 - accuracy: 0.5364
639/1563 [===========>..................] - ETA: 1s - loss: 1.3175 - accuracy: 0.5360
676/1563 [===========>..................] - ETA: 1s - loss: 1.3156 - accuracy: 0.5359
714/1563 [============>.................] - ETA: 1s - loss: 1.3149 - accuracy: 0.5362
752/1563 [=============>................] - ETA: 1s - loss: 1.3169 - accuracy: 0.5353
790/1563 [==============>...............] - ETA: 1s - loss: 1.3157 - accuracy: 0.5360
827/1563 [==============>...............] - ETA: 0s - loss: 1.3144 - accuracy: 0.5355
865/1563 [===============>..............] - ETA: 0s - loss: 1.3147 - accuracy: 0.5352
904/1563 [================>.............] - ETA: 0s - loss: 1.3158 - accuracy: 0.5352
942/1563 [=================>............] - ETA: 0s - loss: 1.3161 - accuracy: 0.5351
980/1563 [=================>............] - ETA: 0s - loss: 1.3178 - accuracy: 0.5345
1018/1563 [==================>...........] - ETA: 0s - loss: 1.3177 - accuracy: 0.5346
1056/1563 [===================>..........] - ETA: 0s - loss: 1.3181 - accuracy: 0.5347
1095/1563 [====================>.........] - ETA: 0s - loss: 1.3164 - accuracy: 0.5352
1133/1563 [====================>.........] - ETA: 0s - loss: 1.3160 - accuracy: 0.5356
1171/1563 [=====================>........] - ETA: 0s - loss: 1.3158 - accuracy: 0.5358
1209/1563 [======================>.......] - ETA: 0s - loss: 1.3154 - accuracy: 0.5368
1247/1563 [======================>.......] - ETA: 0s - loss: 1.3171 - accuracy: 0.5361
1285/1563 [=======================>......] - ETA: 0s - loss: 1.3184 - accuracy: 0.5360
1323/1563 [========================>.....] - ETA: 0s - loss: 1.3174 - accuracy: 0.5364
1360/1563 [=========================>....] - ETA: 0s - loss: 1.3178 - accuracy: 0.5365
1398/1563 [=========================>....] - ETA: 0s - loss: 1.3178 - accuracy: 0.5360
1435/1563 [==========================>...] - ETA: 0s - loss: 1.3174 - accuracy: 0.5358
1472/1563 [===========================>..] - ETA: 0s - loss: 1.3163 - accuracy: 0.5363
1509/1563 [===========================>..] - ETA: 0s - loss: 1.3162 - accuracy: 0.5360
1546/1563 [============================>.] - ETA: 0s - loss: 1.3152 - accuracy: 0.5360
1563/1563 [==============================] - 2s 1ms/step - loss: 1.3146 - accuracy: 0.5362
Epoch 9/10
1/1563 [..............................] - ETA: 2s - loss: 1.0640 - accuracy: 0.5625
38/1563 [..............................] - ETA: 2s - loss: 1.2931 - accuracy: 0.5403
75/1563 [>.............................] - ETA: 2s - loss: 1.2993 - accuracy: 0.5446
113/1563 [=>............................] - ETA: 1s - loss: 1.2887 - accuracy: 0.5509
151/1563 [=>............................] - ETA: 1s - loss: 1.2837 - accuracy: 0.5511
188/1563 [==>...........................] - ETA: 1s - loss: 1.2962 - accuracy: 0.5439
225/1563 [===>..........................] - ETA: 1s - loss: 1.2968 - accuracy: 0.5414
263/1563 [====>.........................] - ETA: 1s - loss: 1.2867 - accuracy: 0.5466
302/1563 [====>.........................] - ETA: 1s - loss: 1.2900 - accuracy: 0.5464
341/1563 [=====>........................] - ETA: 1s - loss: 1.2923 - accuracy: 0.5463
380/1563 [======>.......................] - ETA: 1s - loss: 1.2902 - accuracy: 0.5476
419/1563 [=======>......................] - ETA: 1s - loss: 1.2867 - accuracy: 0.5491
458/1563 [=======>......................] - ETA: 1s - loss: 1.2895 - accuracy: 0.5477
495/1563 [========>.....................] - ETA: 1s - loss: 1.2911 - accuracy: 0.5474
532/1563 [=========>....................] - ETA: 1s - loss: 1.2899 - accuracy: 0.5476
569/1563 [=========>....................] - ETA: 1s - loss: 1.2909 - accuracy: 0.5472
608/1563 [==========>...................] - ETA: 1s - loss: 1.2887 - accuracy: 0.5475
647/1563 [===========>..................] - ETA: 1s - loss: 1.2915 - accuracy: 0.5468
686/1563 [============>.................] - ETA: 1s - loss: 1.2902 - accuracy: 0.5474
725/1563 [============>.................] - ETA: 1s - loss: 1.2914 - accuracy: 0.5463
762/1563 [=============>................] - ETA: 1s - loss: 1.2911 - accuracy: 0.5463
800/1563 [==============>...............] - ETA: 1s - loss: 1.2915 - accuracy: 0.5458
839/1563 [===============>..............] - ETA: 0s - loss: 1.2915 - accuracy: 0.5451
877/1563 [===============>..............] - ETA: 0s - loss: 1.2901 - accuracy: 0.5453
916/1563 [================>.............] - ETA: 0s - loss: 1.2888 - accuracy: 0.5454
955/1563 [=================>............] - ETA: 0s - loss: 1.2883 - accuracy: 0.5460
993/1563 [==================>...........] - ETA: 0s - loss: 1.2873 - accuracy: 0.5464
1031/1563 [==================>...........] - ETA: 0s - loss: 1.2877 - accuracy: 0.5460
1069/1563 [===================>..........] - ETA: 0s - loss: 1.2895 - accuracy: 0.5454
1108/1563 [====================>.........] - ETA: 0s - loss: 1.2894 - accuracy: 0.5457
1146/1563 [====================>.........] - ETA: 0s - loss: 1.2885 - accuracy: 0.5462
1184/1563 [=====================>........] - ETA: 0s - loss: 1.2887 - accuracy: 0.5463
1222/1563 [======================>.......] - ETA: 0s - loss: 1.2872 - accuracy: 0.5470
1261/1563 [=======================>......] - ETA: 0s - loss: 1.2861 - accuracy: 0.5473
1300/1563 [=======================>......] - ETA: 0s - loss: 1.2869 - accuracy: 0.5472
1338/1563 [========================>.....] - ETA: 0s - loss: 1.2871 - accuracy: 0.5470
1376/1563 [=========================>....] - ETA: 0s - loss: 1.2858 - accuracy: 0.5476
1415/1563 [==========================>...] - ETA: 0s - loss: 1.2843 - accuracy: 0.5481
1453/1563 [==========================>...] - ETA: 0s - loss: 1.2847 - accuracy: 0.5477
1490/1563 [===========================>..] - ETA: 0s - loss: 1.2836 - accuracy: 0.5478
1527/1563 [============================>.] - ETA: 0s - loss: 1.2828 - accuracy: 0.5482
1563/1563 [==============================] - 2s 1ms/step - loss: 1.2837 - accuracy: 0.5479
Epoch 10/10
1/1563 [..............................] - ETA: 2s - loss: 1.0096 - accuracy: 0.6250
38/1563 [..............................] - ETA: 2s - loss: 1.2050 - accuracy: 0.5822
76/1563 [>.............................] - ETA: 1s - loss: 1.2122 - accuracy: 0.5859
114/1563 [=>............................] - ETA: 1s - loss: 1.2320 - accuracy: 0.5732
152/1563 [=>............................] - ETA: 1s - loss: 1.2333 - accuracy: 0.5683
190/1563 [==>...........................] - ETA: 1s - loss: 1.2460 - accuracy: 0.5645
228/1563 [===>..........................] - ETA: 1s - loss: 1.2419 - accuracy: 0.5662
267/1563 [====>.........................] - ETA: 1s - loss: 1.2374 - accuracy: 0.5680
305/1563 [====>.........................] - ETA: 1s - loss: 1.2373 - accuracy: 0.5677
342/1563 [=====>........................] - ETA: 1s - loss: 1.2395 - accuracy: 0.5673
379/1563 [======>.......................] - ETA: 1s - loss: 1.2372 - accuracy: 0.5678
416/1563 [======>.......................] - ETA: 1s - loss: 1.2356 - accuracy: 0.5683
454/1563 [=======>......................] - ETA: 1s - loss: 1.2373 - accuracy: 0.5679
493/1563 [========>.....................] - ETA: 1s - loss: 1.2405 - accuracy: 0.5666
532/1563 [=========>....................] - ETA: 1s - loss: 1.2390 - accuracy: 0.5667
570/1563 [=========>....................] - ETA: 1s - loss: 1.2395 - accuracy: 0.5651
608/1563 [==========>...................] - ETA: 1s - loss: 1.2433 - accuracy: 0.5643
646/1563 [===========>..................] - ETA: 1s - loss: 1.2454 - accuracy: 0.5641
683/1563 [============>.................] - ETA: 1s - loss: 1.2471 - accuracy: 0.5636
720/1563 [============>.................] - ETA: 1s - loss: 1.2492 - accuracy: 0.5623
757/1563 [=============>................] - ETA: 1s - loss: 1.2499 - accuracy: 0.5627
794/1563 [==============>...............] - ETA: 1s - loss: 1.2487 - accuracy: 0.5631
831/1563 [==============>...............] - ETA: 0s - loss: 1.2488 - accuracy: 0.5631
868/1563 [===============>..............] - ETA: 0s - loss: 1.2502 - accuracy: 0.5630
905/1563 [================>.............] - ETA: 0s - loss: 1.2512 - accuracy: 0.5629
941/1563 [=================>............] - ETA: 0s - loss: 1.2506 - accuracy: 0.5633
978/1563 [=================>............] - ETA: 0s - loss: 1.2510 - accuracy: 0.5633
1015/1563 [==================>...........] - ETA: 0s - loss: 1.2527 - accuracy: 0.5622
1053/1563 [===================>..........] - ETA: 0s - loss: 1.2512 - accuracy: 0.5622
1090/1563 [===================>..........] - ETA: 0s - loss: 1.2532 - accuracy: 0.5617
1127/1563 [====================>.........] - ETA: 0s - loss: 1.2536 - accuracy: 0.5614
1164/1563 [=====================>........] - ETA: 0s - loss: 1.2523 - accuracy: 0.5622
1201/1563 [======================>.......] - ETA: 0s - loss: 1.2522 - accuracy: 0.5618
1238/1563 [======================>.......] - ETA: 0s - loss: 1.2513 - accuracy: 0.5623
1275/1563 [=======================>......] - ETA: 0s - loss: 1.2512 - accuracy: 0.5624
1312/1563 [========================>.....] - ETA: 0s - loss: 1.2516 - accuracy: 0.5622
1349/1563 [========================>.....] - ETA: 0s - loss: 1.2509 - accuracy: 0.5622
1386/1563 [=========================>....] - ETA: 0s - loss: 1.2507 - accuracy: 0.5622
1424/1563 [==========================>...] - ETA: 0s - loss: 1.2511 - accuracy: 0.5621
1462/1563 [===========================>..] - ETA: 0s - loss: 1.2514 - accuracy: 0.5619
1500/1563 [===========================>..] - ETA: 0s - loss: 1.2518 - accuracy: 0.5620
1538/1563 [============================>.] - ETA: 0s - loss: 1.2524 - accuracy: 0.5617
1563/1563 [==============================] - 2s 1ms/step - loss: 1.2520 - accuracy: 0.5620
Model: "sequential"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
flatten (Flatten) (None, 3072) 0
dense (Dense) (None, 3000) 9219000
dense_1 (Dense) (None, 1000) 3001000
dense_2 (Dense) (None, 10) 10010
=================================================================
Total params: 12,230,010
Trainable params: 12,230,010
Non-trainable params: 0
_________________________________________________________________
None
wall time seconds (without loading test data set)= 25.188053369522095
hydro03
[SSH] completed
[SSH] exit-status: 0
[workspace] $ /bin/sh -xe /tmp/jenkins9765915436923485298.sh
+ scp 'HYDRO_REMOTE:~svchydrojenkins/python/tensorflow_time.txt' /var/lib/jenkins/jobs/tensorflow/workspace
Recording plot data
Saving plot series data from: /var/lib/jenkins/jobs/tensorflow/workspace/tensorflow_time.txt
Finished: SUCCESS
![[Jenkins]](/static/755cd4af/images/svgs/logo.svg){kind=logo}
